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PREFACE 


This  report  describes  the  decision  model  and  associated  process 
support  structure  of  the  Job  Aiding/Training  Allocation 
Technologies,  a  conceptual  methodology  which  aids  decision  makers 
in  determining  whether  and  how  tasks  should  be  instructed  using 
training,  job-aiding,  or  some  combination  of  the  two  methods. 

This  study  was  sponsored  by  the  Air  Force  Human  Resources 
Laboratory  under  Contract  F33615-86-C-0545 .  The  authors  gratefully 
acknowledge  the  many  experts  who  contributed  their  time  and 
expertise  in  our  pursuit  of  their  aiding  and  training  decision 
policies . 
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SUMMARY 


The  objective  of  the  JATAT  Phase  3  project  was  to  further  investigate  the 
task  analysis  and  trade-off  formulation  components  of  the  original  JATAT 
methodology.  This  report  presents  the  resultant  conceptual  decision  model  and 
its  associated  process  support  structure.  Also  included  are  the  results  and 
conclusions  of  two  independent  evaluations  performed  in  support  of  the 
development  effort.  The  first  is  a  statistical  analysis  of  the  decision  attributes 
which  contribute  to  the  aiding/training  decision  process.  The  second  is  an 
informal  investigation  of  the  internal  validity  of  the  newly  formulated  model. 
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INTRODUCTION 


The  ever  increasing  complexity  of  operational  Air  Force  systems  continues  to 
place  greater  demands  on  the  personnel  operating  and  maintaining  them  (AFHRL 
Report,  1986).  The  increased  sophistication  of  these  systems  coupled  with  decreased 
force  levels,  declining  entry-level  skills,  and  the  need  to  limit  military  training  budgets 
are  forcing  Manpower,  Personnel,  and  Training  (MPT)  agencies  to  seek  more  efficient 
methods  of  maintaining  and  improving  operational  readiness  (Boohe^  1978;  Duncan, 
1985).  In  this  environment  of  "doing  more  with  less",  consideration  of  the  issues 
surrounding  job  aiding  and  training  are  paramount. 

One  such  issue  is  the  selection  of  appropriate  aiding  and  training  methods  for  a 
specified  task.  Technical  training,  for  example,  serves  as  the  source  of  knowledge  and 
skills  essential  to  task  performance.  In  other  words,  training  "creates  the  potential  to 
perform"  while  job  aiding,  in  contrast,  directly  augments  the  performance  (Rouse  and 
Johnson,  1989).  Job  aids  collectively  refer  to  those  devices  with  the  capacity  to  store 
and  retrieve  the  "How",  "What",  and  "When"  information  pertinent  to  the  performance  of 
a  particular  task. 

The  process  of  selecting  aiding  and  training  is  not  a  simple  one.  The  same 
technological  advances  which  are  responsible  for  the  increased  sophistication  of 
operational  weapons  systems  are  also  enabling  the  development  of  a  wide  variety  of 
aiding  devices  and  training  techniques.  Selection  from  among  this  array  of  aiding  and 
training  alternatives  is  based  on  a  myriad  of  interdependent  factors  such  as 
performance-related  effectiveness,  development/implementation  costs  and  system 
design  impact. 

Further  complicating  this  selection  process  is  the  fact  that,  in  most  cases,  no  one 
method  alone  completely  satisfies  the  knowledge  and  skill  requirements  demanded  of 
the  operator/maintainer  in  performing  a  specified  task.  In  these  situations,  there  are 
two  equally  viable  outcomes.  First,  the  aiding/training  (A/T)  solution  may  require 
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several  complimentary  devices  and  techniques  employed  in  combination.  Or, 
second,  a  number  of  independent  alternatives  (or  alternative  combinations)  may  be 
equally  responsive  to  the  task  performance  requirements.  This  latter  condition 
generally  necessitates  some  form  of  trade-off  analysis  in  order  to  make  the  final 
selection.  The  formulation  and  evaluation  of  these  A/T  tradeoffs  are  necessary 
components  of  the  decisions  made  by  MPT  analysts,  system  designers,  and  personnel 
supervisors  through-out  the  Air  Force. 

An  example  of  the  considerations  of  such  an  analysis  might  occur  as  follows.  As 
an  information  storage  device,  a  job  aid  facilitates  performance  by  reducing  task 
related  memory  requirements.  This,  in  turn,  reduces  the  training  requirements  for  that 
job  and  generates  the  potential  for  reducing  recurring  resource  expenditures. 
Training,  on  the  other  hand,  can  impart  more  general  knowledge  applicable  to  a 
variety  of  related  tasks.  In  this  case,  the  increased  initial  costs  of  training  a  small, 
multi-disciplinary  work  force  may,  in  the  long-term,  be  offset  by  the  reduced  life-cycle 
costs  of  supporting  a  smaller,  more  flexible  team. 

To  the  extent  that  tradeoffs  analogous  to  this  have  been  addressed  in  the  past,  the 
analyses  have  relied  heavily  on  prior  experience  with  similar  systems.  Typically,  these 
types  of  analyses  have  required  many  person-years  of  effort.  Often,  the  result  has 
been  a  time-consuming  and  expensive  effort  that  provided  insights  which  were  too  late 
to  be  implemented  in  any  substantial  way  (Rouse  and  Johnson,  1989).  Whether  for 
evaluating  current  AFS  job  performance,  selecting  among  new  system  design 
alternatives,  or  ensuring  flight-line  personnel  are  task  qualified,  a  methodology  for 
efficiently  producing  consistent,  timely,  and  supportable  A/T  decisions  is  a  must. 

In  response  to  this  need  the  Armstrong  Laboratory,  Human  Resources  Directorate, 
Brooks  AFB,  has  embraced  a  new  'technology  push'  philosophy  for  developing  future 
A/T  decision  technologies  and  is  sponsoring  the  Job  Aiding/Training  Allocation 
Technologies  (JATAT)  program.  The  purpose  of  JATAT  is  to  develop  a  conceptual 
decision  aiding  methodology  and  a  corresponding  computer-based  decision  support 
system  prototype  designed  to  assist  in  identifying  applicable  A/T  alternatives  and 
evaluating  combinations.  The  expected  benefits  of  such  a  system  include  faster 
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response  times  to  identified  A/T  requirements,  accurate  performance-based 
recommendations,  and  reduced  military  training  costs. 

This  report  is  divided  into  three  sections:  each  section  representing  a  cluster  of 
related  tasks  performed  in  conjunction  with  the  JATAT  project . 

The  first  section  contains  a  short  historical  summary  of  the  JATAT  development 
work  preceding  the  current  phase  of  effort  and  a  review  of  the  essential  components  of 
general  decision  support  systems. 

The  second  section  evaluates  previous  A/T  research  and  describes  the  JATAT 
approach.  The  research  evaluation  addresses  the  strengths  and  weaknesses  of 
several  A/T  efforts  based  on  a  review  of  available  literature.  The  newly  developed 
JATAT  model  is  an  extension  of  the  earlier  proposed  hybrid  methodology  designed  to 
overcome  the  limitations  of  previous  A/T  decision  approaches.  It  provides  greater 
detail  and  a  more  operationally  oriented  approach  to  formulating  /VT  trade-offs. 

And  the  final  section  describes  two  investigative  studies  performed  in  conjunction 
with  the  current  JATAT  model  development.  Study  1  was  a  statistical  survey  of  expert 
A/T  decision  policies.  Its  purpose  was  to  identify  those  attributes  of  A/T  trade-offs 
which  influence  specific  decision  outcomes.  Study  2  was  an  informal  investigation  of 
the  content/construct  validity  of  the  JATAT  model.  This  was  accomplished  through  a 
series  of  semi-structured  interviews  with  A/T  experts  familiar  with  the  JATAT  model. 


A  SYNOPSIS  OF  PREVIOUS  JATAT  DEVELOPMENT 
AND  GENERAL  DECISION  SUPPORT  SYSTEM  CONCEPTS 

Historical  Synopsis 

The  early  part  of  this  effort  was  devoted  to  investigating  existing  computational 
approaches  for  evaluating  trade-offs  between  aiding  and  training  This  initial 
investigation  identified  three  distinctly  different  approaches  to  resolving  AJT  trade-offs: 
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General  Aiding/Training  Decision  Guidelines,  Human/System  Performance 
Predictions,  and  Human/System  Performance  Simulations. 

The  use  of  General  Aiding/Training  Decision  Guidelines  is  based  on  a  compilation 
of  general  knowledge  and  heuristics  for  making  A/T  decisions  extracted  from  the 
cumulative  experience  of  domain  experts  and  experimental  research.  The  trade-offs 
are  implicit  in  the  guidelines.  Thus,  to  a  great  extent,  the  decision  making  is  highly 
proceduralized,  e.g.  if  situation  x,  then  employ  training  type  y  and/or  aiding  type  z 
(Rouse  and  Johnson,  1989). 

A  Human  and  System  Performance  Prediction  approach  involves  predicting 
human/system  performance  as  a  function  of  the  applicable  A/T  alternatives  and  then 
using  these  predictions  as  a  basis  for  comparing  each  of  the  alternatives  in  question. 
This  approach  requires  that  the  relevant  trade-offs  be  explicitly  identified  prior  to  the 
predictive  analysis  (Rouse  and  Johnson,  1989). 

A  Human  and  System  Performance  Simulation  approach,  in  contrast,  involves 
simulating  the  behavior  resulting  from  training  and/or  aiding  which  provides  the  basis 
for  calculating  performance  for  each  trade-off  to  be  evaluated.  This  approach, 
however,  also  requires  explicit  trade-off  formulation  prior  to  the  simulation  execution 
(Rouse  and  Johnson,  1989). 

While  procedurally  complete,  the  use  of  general  A/T  decision  guidelines  lacks  the 
capability  to  pursue  fine-grained  trade-offs,  or  provide  thorough  explanations  for  its 
recommendations.  The  inability  of  this  approach  to  pursue  fine-grained  trade-offs  is 
due  to  the  nature  of  the  guidelines.  For  example,  while  a  satisfactory  A/T  solution  for 
situation  x  may  reside  in  an  available  decision  guideline,  it  is  not  reasonable  to 
assume  that  this  guideline  applies  to  all  possible  deviations  from  x  (i.e.,  x',  x"). 
Therefore,  the  best  a  decision  analyst  can  make  using  this  approach  is  to  select  the 
solution  recommended  for  situation  x.  This  constraint  is  primarily  due  to  the 
unavailability  of  the  decision  rationale  related  to  corresponding  guidelines.  While  the 
rationale  must  exist  in  order  for  the  guideline  to  exist,  it  is  seldom  documented  during 
the  guideline  acquisition  process. 
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Performance  related  approaches,  in  contrast,  can  provide  clear  reasoning  for  their 
solutions,  but,  in  the  absence  of  detailed  A/T  trade-off  formulations,  are  able  to  give 
minimal  decision  aiding  support.  Further,  it  is  often  difficult  to  obtain  the  requisite  data 
for  these  approaches. 

In  response  to  these  findings,  a  hybrid  methodological  framework  was  constructed 
which  combined  the  trade-off  generation  capabilities  of  the  A/T  decision  guidelines 
with  the  reasoning  capabilities  of  the  performance-based  approaches,  in  a 
subsequent  effort  this  framework  was  formalized  into  a  fifteen  step  A/T  decision 
methodology.  The  steps  in  this  methodology  encompass;  data  retrieval,  task  analysis, 
trade-off  formulation,  trade-off  evaluation,  results  interpretation,  and  analysis/decision 
summary. 


The  Decision  Support  System 

There  are  two  fundamental  components  of  a  Decision  Support  System  (DSS)  -- 
the  domain  knowledge  necessary  to  support  the  user’s  decision  process  and  the 
interface  necessary  to  support  the  user's  application  of  this  knowledge.  The 
completeness  of  each  component  is  based  on  the  elements  summarized  in  Table  1. 

There  are  two  types  of  domain  knowledge  associated  with  an  A/T  decision  support 
system;  knowledge  of  the  A/T  decision  process,  and  knowledge  of  the  candidate  task 
classes.  A  task  class  is  a  simple  aggregation  of  tasks  with  a  common  purpose;  e.g., 
trouble-shooting,  preventative  maintenance,  clerical,  etc. 

Each  of  these  knowledge  types  are  essential  to  the  development  of  the  individual 
components  of  a  DSS.  For  example,  knowledge  of  the  A/T  decision  domain 
contributes  to:  1)  the  formulation  of  an  accurate  decision  model  and  process  support 
structure,  2)  the  selection  of  appropriate  A/T  decision  factors,  and  3)  the  specification 
of  expected  levels  of  output  resolu'  -^n.  The  remaining  task  class  knowledge  helps 
provide  the  context-specific  relationships  among  the  A/T  decision  factors  and  the 
values  of  these  factors  in  an  operational  setting. 
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In  accordance  with  the  objective  of  this  phase  of  the  JATAT  effort,  this  report 
focuses  primarily  on  the  knowledge  elements  of  an  A/T  decision  process  model. 


Knowledge  Elements 

Interface  Elements 

Dficision.  Process.  Knowledoe 

0  Representation  of  User's 

0  Decision  Process  Model 

Cognitive  Model  of  the 
Decision  Process 

0  Level  of  Output  Resolution 

0  Support  Functionality 

TasK  Class  Knowledge 

0  Decision  Factors 

0  Relationships  Among  Factors 

0  Factor  Values 

Table  1.  Decision  Support  System  Elements 

The  decision  process  describes  the  set  of  possible  paths  from  problem  definition, 
through  the  intermediate  inputs  and  decisions,  to  the  final  recommendations.  This  can 
be  further  decomposed  into  two  complimentary  structures;  a  process  support  structure 
and  a  general  decision  model.  The  process  support  structure  specifies  the 
appropriate  inputs  to  the  general  decision  model  based  on  the  overall  context  of  the 
decision  problem.  The  general  decision  model,  in  turn,  is  a  network  of  decision  paths 
through  which  intermediate  decision  constructs  are  applied  to  formulate  specific 
solutions. 

It  is  important  to  note  that  the  decision  process  of  the  DSS  need  not  correspond 
identically  to  the  decision  process  of  the  analyst.  It  is  the  role  of  interface  component 
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to  translate  the  dialogue  between  the  user  and  the  DSS  into  the  languages 
recognized  by  each.  For  example,  the  system  requirements,  intermediate  conclusions, 
and  final  recommendations  must  be  presented  to  the  user  in  terms  which  relate  to  the 
user's  perception  of  the  decision  process.  Likewise,  the  user's  inputs  must  be 
consistent  with  the  system's  process  requirements  and  in  a  computationally  usable 
form. 

Decision  factors  are  the  characteristics  of  the  task,  personnel,  equipment,  and 
environment  which  influence  the  outcome  of  the  decision  model.  Selecting  the  factors 
which  are  appropriate  to  a  specific  analysis  is  a  function  of  the  underlying  basis  of  the 
A/T  decision,  the  expected  level  of  output  resolution,  and  the  availability  of  appropriate 
data. 

The  underlying  basis  of  an  A/T  trade-off  decision  can  either  be  based  on  human- 
system  performance  attributes  or  operational  constraints.  For  example,  an  approach 
based  on  performance  attributes  will  emphasize  factors  such  as  personnel 
experience,  task  frequency,  and  equipment  complexity.  In  contrast,  decisions  based 
on  operational  constraints  will  emphasize  cost  and  resource  availability. 

Relationships  among  decision  factors  are  represented  as  intermediate  decision 
paths  throughout  the  general  decision  model.  Collectively,  each  of  the  intermediate 
decision  points  and  their  respective  paths  form  a  contextual  map/network  of  the  overall 
problem  space.  The  path  chosen,  therefore,in  any  given  analysis  will  represent  the 
specific  context  of  the  operational  environment  in  question.  The  characteristics  of 
these  relationships  are  essential  to  the  choice  of  the  decision  model  employed  and, 
hence,  must  be  understood  early  in  the  decision  process. 

The  values  associated  with  each  of  the  selected  factors  provide  the  context- 
specific  data  which  guides  selection  of  the  decision  paths.  Two  issues  directly  related 
to  these  decision  factor  values  include;  1)  the  availability  of  the  supporting  data,  and  2) 
the  intended  level  of  resolution  of  the  process.  Data  availability  can  be  further 
categorized  into  three  distinct  levels:  1)  that  which  is  readily  available  in  a  directly 
usable  form,  2)  that  which  requires  some  degree  of  pre-processing  (i.e.,  transformation 
in  terms  of  abstraction  or  aggregation),  and  3)  that  which  requires  new  or  modified 
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data  acquisition  procedures.  A  high  level  of  resolution  requires  quantitative  input  data, 
while  a  low  level  of  resolution,  on  the  other  hand,  requires  only  qualitative  inputs. 

The  level  of  resolution  of  a  DSS  defines  the  level  of  detail  of  both  the  system 
inputs  and  system  outputs.  For  example,  input  decision  factors  can  exist  at  various 
levels  of  precision  (i.e.,  aptitude  ratings  of  high,  medium,  and  low  versus  quantitative 
ASVAB  scores).  Similarly,  system  A/T  outputs  can  be  represented  as  either  broad 
classes  (e.g.,  train  vs.  aid),  categories  within  classes  (e.g.,  passive  training  techniques 
vs  procedural  aids),  or  instances  of  categories  (classroom  lecture  vs  flip-chart 
checklist).  The  intended  level  of  resolution  will  have  a  major  impact  on  the  extent  of 
the  decision  network  contained  within  the  general  decision  model. 

REVIEW  OF  PREVIOUS  RESEARCH  AND 
DESCRIPTION  OF  THE  JATAT  MODEL 

The  results  of  this  work,  prior  to  this  report,  include:  1)  a  comprehensive  review  of 
current  A/T  literature,  and  2)  the  formulation  of  a  JATAT  decision  model.  The  review 
found  on  the  following  pages  discusses  the  strengths  and  weaknesses  of  current  A/T 
research  and  several  current  A/T  decision  models.  The  JATAT  approach  describes 
the  current  JATAT  decision  process  support  structure  and  the  general  decision  model. 
The  discussion  addresses  each  stage  of  the  process  support  structure  in  detail  and 
describes  the  role  of  the  general  decision  model  at  each  appropriate  juncture. 

Review  of  Available  Aidlna/Tralnina  Literature 

A  review  of  available  A/T  literature  indicates  that  the  majority  of  early  research 
concerning  A/T  decision-making  focused  on  context-free  or  limited  context 
applications.  Further,  the  majority  of  decision  guidelines  produced  by  these  efforts 
related  specific  A/T  solutions  to  specific  decision  factors,  without  regard  to  intermediate 
relationships.  As  indicated  earlier,  decision  models  devoid  of  context  and 
intermediate  decision  relationships  are  likely  to  have  limited  application  to  real-world 
problems. 
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Additionally,  the  models  generated  by  this  previous  research  have  generally 
employed  only  linear  statistical  models  or  deterministic,  decision  tree  approaches. 
Statistical  approaches  attempt  to  quantify  the  relationships  between  the  independent 
and  dependent  variables  associated  with  the  decision  (Irvin,  Blunt,  and  Lamb,  1988). 
In  contrast,  a  decision  tree  approach  represents  a  series  of  empirically  based  rules 
defining  the  relationships  among  known  input  values  and  expected  outcomes 
(Booher,  1978,  Goins,  Marshal,  and  Levine,  1989).  A  third  potential  approach  is  to 
apply  a  knowledge-based  system.  This  approach  employs  rules/heuristics  which 
express  relevant  relationships  and  solutions  to  a  number  of  related  problems  based 
on  expert  experience  and  empirical  research. 

One  method  for  comparatively  evaluating  these  approaches  is  to  ordinally  rank  the 
degree  of  explanation  they  can  provide  pertaining  to  a  particular  decision  analysis. 

A  linear  statistical  approach,  for  example,  employs  weighting  values  as  a  basis  for 
defining  the  relationships  between  variables  and  the  recommended  A/T  solutions. 
The  explanation  available  in  this  approach  for  making  these  recommendations  is 
inherent  in  the  progressive  refinement  and  use  of  statistical  techniques  such  as 
discriminant  function  analysis,  factor  analysis,  principal  component  analysis,  cluster 
analysis,  or  other  similar  multivariate  procedures.  Through  these  progressive 
iterations  the  researcher  eliminates  or  modifies  variables  which  do  not  contribute  to 
deriving  the  A/T  solution.  This  "model"  can  then  be  used  to  determine  the  A/T 
recommendations  for  tasks  for  which  a  user  can  provide  the  requisite  data.  As 
indicated  earlier,  the  rationale  for  recommending  a  specific  N1  solution  set  is  implicit 
in  the  model  and  not  directly  accessible  to  the  user.  Therefore,  in  order  to  offer  the 
task  by  task  explanation  most  useful  to  user  of  a  JATAT  decision  model  it  would  be 
additionally  necessary  to  generate  a  semantic  interpretation  of  the  specific  weights 
and/or  functions  as  they  apply  to  an  individual  task. 

The  empirically  derived  rules  of  a  decision  tree  represent  a  highly  rigid  application 
of  a  predetermined  reasoning  process.  Although  a  decision  tree  approach  may 
frequently  employ  correlational  values  to  guide  certain  intermediate  level  decisions 
the  capability  exists  to  describe  the  decision  path  taken  to  arrive  at  the  recommended 
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solution.  This  increased  degree  of  available  explanation  is,  however,  still  constrained 
by  the  inflexibility  of  the  predetermined  reasoning  process. 


The  expert  rules  of  a  knowledge-based  approach,  in  contrast,  enable  reasoning 
about  the  "How's"  as  well  as  the  "Why's"  of  the  decision  process.  The  knowledge 
employed  by  this  type  of  approach  includes  both  specific  relationships  and 
intermediate  decisions  as  well  as  supporting  rationale  and  other  general  domain- 
related  heuristics.  It  is  likely  that  the  relationships  among  the  decision  factors, 
intermediate  constructs  and  recommended  solutions  would  be  based  on  empirically 
derived  correlational  values.  Knowledge  of  the  supporting  rationale  provides  an 
unparalleled  explanatory  capability  for  accompanying  recommended  solutions. 
Additionally,  the  inclusion  of  other  general  domain-related  heuristics  enables  a  much 
more  flexible  response  to  an  A/T  decision  problem. 

General  Conclusions 

The  following  paragraphs  summarize  the  issues  and  observations  identified  and 
generated  during  our  review  of  the  current  A/T  literature. 

1.  The  majority  of  efforts  in  this  area  have  only  considered  the  decision  at  the  level 
of  training  and/or  aiding.  Only  Booher  (1978)  attempts  to  specify  particular  types  of 
training  or  aiding  for  given  task-  and  personnel-related  conditions. 

2.  Most  articles  indicate  whether  to  train  or  aid  if  a  particular  condition  is  satisfied, 
but  not  what  to  do  if  the  condition  is  not  satisfied.  Choice  of  the  alternative  option  is, 
therefore,  either  not  as  clearly  defined  and  left  to  the  analyst's  discretion,  or  not  fully 
considered  as  part  of  the  available  literature.  For  example,  while  it  is  stated  that  tasks 
with  high  information  quantity  should  be  aided,  it  is  never  explicitly  stated  what  should 
be  applied  to  tasks  with  low  information  quantity.  Figure  1  is  a  sampling  of  several 
decision  factors  addressed  in  the  current  A/T  literature  in  which  this  phenomenon  is 
manifested.  (The  appropriate  literature  source  is  sited  in  each  factor  box.)  The  solid 
lines  represent  those  relationships  explicitly  identified  in  the  literature.  The  dashed 
lines  represent  extrapolations  which  have  no  empirical  basis. 


-  Solid  lines  indicate  relationships  drawn  directly  from  specifie< 
literature. 

•  Dashed  lines  indicate  extrapolations. 


Figure  1.  Factor-Solution  Relationships  from  Previous  Research 


3.  Another  issue  indirectly  presented  in  Figure  1  is  the  lack  of  specified 
relationships  among  input  variables.  This  situation  can  frequently  result  in  conflicting 
recommendations.  For  example,  how  is  the  decision  resolved  if  error  consequence  is 
low  (indicating  "train")  but  frequency  of  performance  is  infrequent  (indicating  "aid")? 

4.  Figure  2,  a  graphic  representation  of  the  20  A/T  decision  rules  specified  by 
Goins,  Marshal,  and  Levine  (1989),  addresses  the  above  issue  by  revealing  the 
potential  complexity  of  relationships  among  variables.  For  example.  Rule  #1 1  of  Goins 
et.  al.  (1989)  states  that  "If  an  equipment  hazard  does  not  exist,  and  the  task  is  critical, 
and  the  performance  frequency  is  less  than  3  months,  then  train". 

5.  While  the  representation  in  Figure  2  shows  an  early  attempt  to  address  the 
complexity  of  factor  relationships,  this  representation,  like  others,  fails  to  address  the 
possibility  of  A/T  trade-offs.  In  fact,  the  majority  of  current  A/T  literature  avoids  those 
cases  in  which  a  combination  of  aiding  and  training  should  be  employed.  This, 
unfortunately,  constitutes  a  large  portion  of  A/T  decision  situations.  For  instance,  it  is 
highly  likely  that  a  task  with  high  error  consequence  will  be  trained  as  well  as  aided. 

6.  Conceptual  and  operational  definitions  are  frequently  unspecified.  And,  in 
those  cases  in  which  the  variables  are  specified,  they  are  frequently  internally 
ambiguous  and  ill-defined,  and  inconsistent  with  the  definitions  of  similar  factors  in 
other  literature.  For  example,  how  many  is  "many"  and  how  few  is  "few"  training 
resources?  (See  Appendix  A) 

7.  Finally,  the  rationale  for  the  decisions  are  not  explicitly  documented. 

Our  conclusion  is  that  no  one  model  or  approach  completely  satisfies  the  JATAT 
DSS  requirements.  Hence,  a  hybrid  model  which  incorporates  the  strengths  of  each 
of  these  models,  where  appropriate.  Is  necessary.  This  is  the  role  of  the  general 
decision  model  and  process  support  structure  described  in  the  following  sections. 
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Figure  2.  Variable  Relationship  Netwoil^  --  Goins,  Marshal,  and  Levine  (1 989) 


Itie_JATAT  Approach 


The  first  step  in  developing  any  decision  model  is  to  specify  the  context  and 
boundaries  of  the  problem  space.  In  accordance  with  the  focus  of  this  phase  of  the 
JATAT  effort  the  problem  is  practically  bounded  by,  and  the  representation 
correspondingly  structured  after,  the  first  seven  steps  of  the  original  fifteen  step 
methodology.  These  include: 


•  Understand  the  Job 

•  Decompose  via  a  Task  Taxonomy 

•  Assess  Human  Limitations,  Abilities,  and  Preferences 

•  Map  Limitations,  Abilities,  and  Preferences  to  a  Taxonomy  of  Training 
Alternatives 

•  Map  Limitations,  Abilities,  and  Preferences  to  a  Taxonomy  of  Aiding 
Alternatives 

•  Make  Obvious  Choices 

•  Coalesce  Interdependent  Trade-offs 

•  Choose  Measures  of  Performance 

•  Choose  Input-Output  Representations 

•  Identify  Requisite  Structure  and  Parameters  for  Representations 

•  If  Necessary,  Represent  Learning  Process 

•  Apply  Methods  of  Analysis  to  Representations 

•  Interpret  Results 

•  Compile  Assumptions  and  Consequences  of  Trade-offs 

•  Form  Sets  of  Trade-offs  with  Consistent  Assumptions  and  Consequences 

Regarding  Personnel  _ 

Table  2.  Fifteen  Steps  of  Original  JATAT  Methodology 

The  culmination  of  this  decision  model  development  is  represented  in  the  two  part 
process  support  structure  diagram  in  Figures  3  and  4  and  the  general  decision  model 
to  be  discussed  later.  The  process  support  structure  is  responsible  for  providing  the 
operational  context  for  the  decision  model  and  for  guiding  the  task 
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Figure  3.  The  JATAT  Process  Support  Structure  (Part  1) 


Evakuto 


The  JATAT  Process  Suppon  Struture 


analysis/decomposition  process  which  provides  the  inputs  to  the  general  decision 
model. 


Part  One  (Figure  3)  of  the  process  support  structure  describes  an  operationally 
oriented  task  analysis  process.  The  architecture  of  this  process  is  based  primarily 
upon  the  availability  of  data  from  current  Air  Force  data  sources  (i.e.,  the  Occupational 
Research  Data  Bank,  and  the  more  recent  AFS  Task  Analyses  documents).  Part  Two 
(Figure  4)  describes  an  attribute/performance  based  analysis  of  the  knowledge  and 
skills  derived  from  the  preceding  task  analysis.  These  two  parts  can  be  further  divided 
into  the  following  five  process  segments: 

•  The  Decision  Process  Entry  Points 

•  Activity,  Knowledge,  and  Skill  Identification 

•  Relevant  Knowledge  and  Skill  Specification 

•  Skill  Analysis 

•  Knowledge  Analysis  and  Aiding/Training  Selection 
The  Decision  Entry  Points 

There  are  essentially  three  potential  reasons  for  employing  the  JATAT  decision 
process.  First,  a  new  system  is  being  developed  which,  when  implemented,  will 
require  operators  and  maintainers  to  perform  a  series  of  new  tasks.  In  this  case,  each 
of  the  new  tasks  would  be  independently  evaluated  using  the  JATAT  DSS.  Second, 
current  performance  on  a  particular  task  within  the  operational  environment  is 
unsatisfactory.  This  task  would  be  evaluated  to  identify  potential  sources  of 
deficiencies  and  alternative  approaches  to  aiding  or  training.  And  finally,  an  Air  Force 
MPT  organization  is  investigating  opportunities  for  reduced  training  resource 
expenditures.  Candidate  tasks  would  be  analyzed  through  JATAT  to  determine 
alternative  methods  for  augmenting  or  enhancing  performance.  These  alternat'ves 
can  then  be  comparatively  evaluated  on  a  cost  basis. 

The  shaded  portion  of  Figure  5  represents  the  decision  entry  points  of  the  process 
support  structure.  This  same  shading  scheme  will  be  used  in  Figures  6  -  9  to  indicate 
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Figure  5.  The  Decision  Entry  Points 


those  components  of  the  process  support  structure  which  comprise  each  of  the  four 
remaining  process  segments. 

Activity.  Knowledge,  and  Skill  Identification 

A  thorough  task  decomposition  of  this  nature  can  be  difficult  but  is  a  necessary 
component  of  any  task  related  decision  process.  The  current  decomposition  process 
is  based  on  the  activities,  knowledge  and  skills  related  to  the  task  in  question. 
Specifically,  each  task  is  composed  of  a  number  of  related  (either  temporally, 
functionally,  or  otherwise)  activities.  Each  activity  has  associated  with  it  several 
knowledge  and  skill  components.  The  goal  of  this  decomposition  process  is  to  identify 
all  the  related  activities,  knowledge  and  skills  pertinent  to  the  task.  (See  Rgure  6.) 

This  identification  process  quickly  becomes  intractable  if  the  analyst  is  unfamiliar 
with  the  domain  in  question.  Fortunately,  recent  OSR  efforts  have  begun  to  compile 
activities,  knowledge  and  skills  required  for  each  of  the  task  responsibilities  of  a  given 
Air  Force  Specialty  (AFS).  (Ref:  454X0  Task  Analysis).  Task  decomposition  within  the 
JATAT  decision  process  model  is  based  upon  these  OSR  analyses.  As  mentioned  in 
an  earlier  JATAT  report  (Zenyuh,  Frey,  Rouse,  and  Lamb,  1990)  consistent  definitions 
of  tasks,  activities,  knowledge,  and  skills  are  requisite  to  the  useful  application  of  a 
decision  support  system. 

These  requisite  knowledge  and  skills  define  the  minimum  proficiency 
requirements  for  each  activity  and  task  performance.  Minimum  proficiency,  in  this 
case,  indicates  that  the  operator/maintainer  can  satisfy  the  minimum  performance 
requirements  for  the  defined  task.  Current  Air  Force  training  procedures  attribute  no 
additional  benefit  to  performance  capabilities  beyond  the  minimum  requirement. 


Relevant  Knowledge  and  Skills  Specification 

Once  the  task  decomposition  is  complete  and  all  the  related  activities,  knowledge 
and  skills  are  identified,  it  is  necessary  to  select  the  subset  of  knowledge  and  skills 
which  is  to  be  analyzed.  (See  Figure  7.)  The  purpose  of  this  selection  process  is  to 
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Figure  7.  Specificatior)  of  Relevant  Knowledge  and  Skills 


eliminate  those  knowledge  and  skills  for  which  successful  training  programs  already 
exist.  More  specifically,  a  decision  maker  must  determine:  1)  if  any  of  these 
requirements  are  satisfied  as  part  of  the  training  program  for  the  same  specialty  -  for 
other  tasks,  or  2)  if  any  of  these  requirements  are  satisfied  as  part  of  the  training 
program  for  a  different  specialty  -  same  task.  Conceivably,  this  could  be  assisted  by 
employing  an  automated  pattern  matching  algorithm  to  compare  these  knowledge  and 
skills  between  related  databases. 

This  process,  as  currently  envisioned,  would  rely  heavily  upon  existing  operational 
AFS  training  programs  --  each  particular  program  having  specific  knowledge  and  skills 
which  are  expected  to  be  acquired  by  the  students  during  their  training.  For  example, 
Jet  Engine  Maintenance  personnel  are  already  able  to  "Recall  Power  Supply 
Handling  Procedures"  as  a  result  of  their  initial  aiding  and  training.  Therefore,  it  would 
be  unnecessary  to  perform  an  in  depth  A/T  analysis  to  determine  how  best  to  enable 
Avionics  Maintenance  personnel  to  perform  a  similar  function. 


Skill  Analysis 

Skill  analysis  is  based  on  further  decomposition  of  their  physical  and  cognitive 
components.  (See  Figure  8.) 

The  physical  requirements  of  a  task  can  easily  be  evaluated  against  the  known 
physical  capabilities  of  the  personnel  (e.g.,  height,  weight,  strength,  etc.).  If 
deficiencies  exist,  the  decision-maker  has  the  options  of  either  recommending  the  use 
of  a  tool  to  augment  performance  or  re-establishing  the  qualification  criteria  for 
personnel  selection.  If  the  personnel  are  physically  qualified,  the  analyst  can 
recommend  instruction  of  the  desired  performance  behavior. 

Cognitive  requirements  are  simply  analyzed  as  additional  requisite  knowledge 
components. 
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Figured.  Skill  Analysis 


Once  the  list  of  knowledge  components  to  be  analyzed  is  established,  the 
decision-maker  is  confronted  with  a  series  of  evaluations  (decision  points),  each  with 
the  purpose  of  more  narrowly  focusing  the  problem  space.  Each  subsequent  decision 
is  directed  at  eliminating  another  portion  of  the  problem  space.  The  ultimate  result 
being  the  determination  of  the  most  appropriate  A/T  solution(s)  for  the  knowledge 
requirement.  The  Knowledge  Analysis  and  Aiding/Training  Selection  segment  of  the 
process  structure,  highlighted  in  Figure  9,  represents  this  decision  flow  process. 

It  is,  as  yet,  unclear  whether  each  knowledge  component  must  be  independently 
analyzed  for  A/T  alternatives  or  if  they  can  be  aggregated  across  some  underlying 
characteristic  and  analyzed  as  a  group.  While  this  issue  is  worthy  of  further  research, 
it  is  beyond  the  scope  of  the  current  effort. 

The  first  decision  in  this  segment  of  the  process  structure  is  to  assess  the  potential 
"barriers  to  aiding".  The  objective  at  this  point  is  to  determine  if  it  is  at  all  possible  to 
efficiently  aid  any  component  of  the  required  knowledge.  (This  is  not  an  issue  of 
whether  or  not  an  applicable  aid  can  completely  satisfy  the  knowledge  requirements  -- 
this  issue  is  addressed  at  a  subsequent  decision  point) 

The  rationale  for  making  this  determination  first  is  that  aiding  is  generally  more 
cost  effective  than  training.  Therefore,  the  philosophy  is  simply  to  "aid  if  you  can".  This 
is  further  evidenced  in  the  decision  flow  diagram  in  that  the  aiding  related  decisions 
are  always  resolved  prior  to  addressing  training  related  decisions. 

Barriers  to  aiding  define  a  specific  cluster  of  decision  factor  values  (e.g.,  high  task 
performance  rate)  which  prevent  the  useful  application  of  known  aiding  approaches  to 
a  particular  knowledge  and  task.  These  can  be  categorized  as  those  which  prevent 
aiding  for  physical  (e.g.,  limited  space),  technical  (e.g.,  aid  technology  insufficient), 
temporal  (e.g.,  high  task  rate  prohibits  referencing  aid),  or  psycho-social  reasons  (e.g., 
credibility  of  performer  degraded  by  existence  of  aid). 
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Knowledge  Analysis  and  AkJing/T raining  Selection 


Figure  10  represents  a  time  based  model  of  a  potential  barrier  to  aiding.  In  this  case, 
the  time  required  to  perform  the  task  exceeds  the  time  available  and  a  task  redefinition 
is  recommended. 

Given  that  some  barrier(s)  to  aiding  exists,  (i.e.,  there  is  no  type  of  aiding  which 
can,  to  some  level,  satisfy  the  knowledge  requirement)  then  training  must  be 
considered  as  the  only  feasible  alternative.  If  training  alone  can  completely  satisfy  the 
target  knowledge  requirements,  then  candidate  approaches  can  be  considered. 
Where  training,  as  such,  is  not  feasible,  the  task  must  be  either  automated  or 
redesigned. 

On  the  other  hand,  if  aiding  is  feasible,  then  it  must  be  determined  if  an  aiding 
approach{es)  (or  combinations,  thereof),  which  can  fully  satisfy  the  target  knowledge 
requirements,  exists.  If  so,  then  candidate  aiding  approaches  can  be  considered. 
(Note  that,  in  this  case,  the  decision-maker  may  also  need  to  address  the  potential 
training  requirements  of  the  selected  aiding  approaches.) 

Assuming  that  uic'  .ig  is  not  prevented  by  any  known  decision  factors,  but  will  not 
singularly  satisfy  the  knowledge  requirement,  training  should  be  considered  as  a 
component  of  the  solution.  If,  at  this  point,  there  are  any  physical,  technical,  temporal, 
or  psycho  social  barriers  to  training,  the  task  is  intractable  from  an  A/T  perspective  and 
must  be  automated  or  redesigned.  Otherwise,  candidate  A/T  combination  solutions 
can  be  considered. 

It  is  important  to  notice  that,  in  each  case,  the  A/T  recommendations  are  based 
purely  on  performance  attributes.  Operational  issues,  such  as  development  and  life- 
cycle  cost  and  geographical  availability,  while  essential  to  a  complete  evaluation,  are 
better  analyzed  using  other  MPT  models  (e.g.,  TDS). 
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Barriers  to  Aiding 
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Figure  1 0.  Barriers  to  Aiding  --  A  Temporal  Model 


The  simplicity  of  the  Knowledge  Analysis  and  Aiding/Training  Selection  segment 
of  the  decision  process  is  deceiving.  In  actuality,  each  decision  point  represents  a 
complex  confluence  of  knowledge  characteristics,  task-,  personnel-,  equipment-,  and 
environmental-factors  and  their  underlying  interdependencies.  These  factors  and 
interdependencies,  as  well  as  the  candidate  A/T  solutions,  form  the  basis  for  the 
particular  decision  model  employed  at  each  respective  point.  These  individual 
decisions  can  then  be  aggregated  into  a  composite  model  which  represents  the 
boundaries  and  context  of  the  entire  problem  space.  For  the  purpose  of  this  report,  the 
composite  model  developed  as  part  of  this  effort  will  be  referred  to  at  the  JATAT 
General  Decision  Model. 

As  mentioned  earlier  in  this  report,  the  characteristics  of  these  factors  and  their 
relationships  greatly  influence  the  architecture  of  a  model  used  to  represent  a  decision 
process.  For  the  current  A/T  decision  model,  the  most  formative  characteristic 
influencing  its  design  is  the  large  number  of  task-related  factors  and  their  inter¬ 
relationships. 

This  condition  creates  several  problems.  The  first  obvious  difficulty,  is 
determining  how  each  of  the  numerous  variables  should  affect  the  decisions  in 
general,  and  how  such  relationships  are  affected,  in  particular,  by  differing  contexts. 
Secondly,  if  all  possible  interactions  are  considered,  data  collection  requirements 
become  intractable  --  yet,  if  data  collection  is  compromised,  parameter  estimates  have 
wider  ranges  of  uncertainty  and  predictions  are  subject  to  larger  errors.  Finally,  even  if 
relationships  can  be  identified  and  parameters  appropriately  estimated,  without  some 
form  of  intervening  assistance,  the  whole  approach  may  be  rather  opaque  since  the 
user  may  have  difficulty  intuiting  which  factors  are  playing  what  roles  in  the  decision 
process. 
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One  approach  is  to  employ  intermediate  constructs/classifications  which  map  the 
input  variables  to  the  proposed  solutions  and  represent  the  entire  decision  process  as 
a  network  of  these  intervening  relationships  (Chandrasekaran  and  Goel,  1988). 

The  current  JATAT  general  A/T  decision  model  employs  one  such  approach.  (See 
Figure  11.)  Task-related  factors  are  represented  as  inputs  and  constraints,  candidate 
A/T  solution  sets  are  represented  as  outputs,  and  the  decision  paradigm  is 
represented  as  a  reasoning  process  based  on  intermediate  constructs. 

Addressing  the  model  in  detail,  each  of  the  task,  personnel,  equipment  and 
environmental  characteristics/factors  place  various  resource  demands  on  the 
individual  performing  that  task.  These  demands,  in  turn,  have  certain  implications  for 
the  learning  and/or  task  performance  capabilities  of  the  personnel.  Potential  sources 
of  resolution  for  these  implications  exist  in  the  form  of  general  aiding  or  training 
approaches.  Once  an  A/T  approach(es),  which  satisfactorily  responds  to  these 
implications,  has  been  identified,  a  specific  aiding  device  or  training  technique  can  be 
selected. 

This  representation:  input  data  to  abstract  requirements,  abstract  requirements  to 
abstract  solutions,  and  finally  abstract  solutions  to  a  specific  solution(s),  portrays  the 
pattern  of  intermediate  inferences  associated  with  the  decision  process.  This  is 
commonly  referred  to  as  an  inference  structure  representation  (Steels,  1990). 

There  are  three  advantages  to  applying  an  inference  structure  approach  to  the 
general  A/T  decisions  model.  One,  it  presents  the  underlying  structure  of  the  A/T 
decision  rules  and  heuristics.  This  permits  subjective  evaluation,  by  other  domain 
experts,  of  the  reasoning  underlying  the  variable  relationships,  implicit  in  the 
intermediate  constructs.  Two,  it  shows  the  relationships  among  the  rules/heuristics. 
The  benefit  of  this  manifestation  is  two-fold.  First,  the  knowledge  acquisition  process 
can  be  more  accurately  constrained  to  the  domain  knowledge  of  interest.  And  second, 
it  enables  an  automated  system  to  provide  explanations  for  its  recommendations. 
Third,  the  same  structure  can  be  applied  to  widely  diverse  domains  and  tasks  (Steels, 
1990). 
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As  indicated  above,  the  general  decision  model  is  a  composite  of  each  of  the 
individual  decision  points  in  the  Knowledge  Analysis  and  Aiding/Training  Selection 
segment  of  the  process  structure.  It  is  not  contained  in  its  entirety  in  any  one  decision 
or  selection.  Particular  values  and  constructs,  though,  are  considered  indispensable 
components  of  each  of  these  decision  points.  The  following  chart  identifies  those 
components  of  the  general  decision  model  which  apply  to  each  of  the  decision  and 
selection  points  in  the  process  model.  (See  Figure  12.) 

While  each  successive  decision  point  contains  progressively  more  of  the  general 
decision  model,  this  does  not  imply  that  the  entire  model  is  iteratively  evaluated. 
Instead,  the  results  of  intermediate  analyses  are  carried  through  to  successive 
decision  points.  For  example,  determining  if  a  knowledge  deficit  can  be  sufficiently 
trained  implies  that  earlier  analyses  of  performance  implications  indicated  aiding,  as  a 
complete  solution,  to  be  infeasible.  In  some  cases,  however,  subsequent  analyses 
may  address  input  factors  not  previously  considered.  For  instance,  while  information 
retention  requirements  may  not  be  considered  among  the  Barriers  to  Aiding,  they  may 
be  addressed  as  a  potential  Barrier  to  Training.  In  these  cases,  the  necessary 
components  of  the  general  model  must  be  re-activated  --  at  least  for  the  new  factor(s). 

Level  of  Resolution 

Implicit  in  the  representation  of  the  general  decision  model  is  the  issue  of  the 
appropriate  level  of  resolution  of  JATAT  decision  support.  At  one  extreme,  an  A/T 
solution  set  which  only  includes  aid  or  train  provides  insufficient  detail  to  adequately 
address  combinations  and  alternatives.  The  recommendations  provided  by  such  a 
model  would  be  devoid  of  alternative  aiding  and  training  approaches.  This  would 
eliminate  the  potential  for  trade-offs  and,  hence,  not  satisfy  one  of  the  objectives  of  this 
effort.  Conversely,  there  is  little  need  for  extreme  detail  (i.e.,  class  size,  course  length, 
etc.)  since  such  information  is  most  useful  to  curriculum  developers  and  aid  designers 
once  an  A/T  need  has  been  identified. 

The  set  of  solutions  shown  in  Figure  1 1  strikes  a  balance  between  these  extremes 
by  allowing  for  the  possibility  of  several  satisfactory  combined  solutions.  This  level  of 
resolution  of  the  candidate  aiding  methods  and  training  environments  can  ;  1)  provide 
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useful  input  to  other  A/T  models  (i.e.,  TDS,  LCOM,  etc.)  and  related  domain  experts, 
and  2)  permit  the  direct  translation  of  a  recommended  solution  to  the  pre-determined 
set  of  Air  Force  technical  training  locations  (i.e.,  Residence  School,  Field  Training 
Detachment,  and  On-the-Job). 


SuDDortinq  Networks 

While  this  decision  model  can  resolve  A/T  issues  in  general,  it  is  difficult  to 
imagine  a  single  instantiation  of  this  network  providing  A/T  recommendations  tailored 
to  specific  contexts.  Therefore,  in  order  to  fully  exploit  the  model's  potential 
functionality,  it  is  necessary  to  compile  multiple  network  versions  with  input  variables 
and  variable  relationships  tailored  to  the  context  of  a  specific  analysis. 

Following  this  train  of  thought,  the  work  accomplished  to  date  indicates  that  the 
variables,  relationships,  and  parameters  of  each  networks  should  be  dependent  upon 
the  class  of  task  being  analyzed  (e.g.,  maintenance  task  analyses  will  differ  from 
administrative  task  analyses).  More  specifically,  the  parameters  (e.g.,  empirical  values 
generated  with  a  linear  statistical  approach)  of  relationships  for  troubleshooting 
maintenance  tasks  are  likely  to  differ  from  those  for  the  non-troubleshooting 
maintenance  tasks.  Therefore,  each  task  type  would  have  associated  with  it  a  network 
tailored  via  a  template  of  corresponding  variables,  relationships,  and  parameters. 

The  precise  representation  of  these  networks  and  corresponding  templates  are  yet 
to  be  determined.  They  are,  however,  likely  to  involve  a  mix  of  linear  statistical 
models,  rule-based  heuristics,  etc. 

JATAT  PHASE  3  EMPIRICAL  STUDIES 

The  objective  of  Phase  3  was  to  further  investigate  the  task  analysis  and  trade-off 
formulation  components  of  the  original  methodology.  Included  in  this  effort  was  the 
development  of  a  model  for  selecting  alternative  A/T  approaches.  The  two  primary  foci, 
therefore,  were  the  A/T  decision  factors  which  most  heavily  influence  alternative 
selection  and  trade-off  formulation,  and  the  representations  which  best  describe  the 
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important  relationships  among  these  factors.  More  specifically,  during  this  phase  of 
effort  we  pursued  answers  to  the  following  factor-related  questions; 

•  What  are  the  important  factors  in  an  A/T  decision? 

•  How  are  they  operationalized? 

•  What  do  we  know  about  the  relationships  among  factors? 

•  What  don't  we  know? 

Another,  equally  important,  issue  to  be  resolved  is  the  internal  validity  of  the  new 
JATAT  model.  In  other  words,  "What  is  the  best  representation  of  the  JATAT  decision 
process  and  the  relationships  among  the  factors?"  Representations  gleaned  from 
human  performance  studies?  Linear  statistical  models?  Or,  heuristic  rules? 

This  section  describes  two  complementary  information  gathering  activities 
designed  to  specifically  address  these  questions;  a  statistical  evaluation  of  expert  A/T 
decision  policies,  and  a  series  of  semi-structured  interviews. 


Study  IL  A  Statistical  Evaluation  of  Expert 
Aidinq/Trainina  Decision  Policies 

Previous  quantitative  analyses  of  expert  A/T  decisions  (Irvin,  Blunt,  and  Lamb, 
1988)  investigated  linear  relationships  among  13  A/T  decision  attributes  and  an 
expert's  recommendation  to  aid,  train,  aid  and  train,  or  neither  for  that  task.  Study  1  of 
this  effort  employed  a  similar  analytical  paradigm  with  three  modifications;  a  different 
set  of  13  decision  attributes  were  analyzed,  several  intermediate  relationships  were 
investigated,  and  a  larger  number  of  decision  alternatives  were  evaluated. 

Decision  Attributes 

The  decision  attributes  analyzed  in  Study  1  represent  a  subset  of  those  extracted 
from  available  A/T  literature.  (See  Appendix  A.)  It  was  apparent  in  this  original 
compilation,  however,  that  there  were  a  large  number  of  discrepancies  (i.e., 
duplications,  ambiguities,  etc.)  among  the  attribute  nomenclatures  and  definitions. 
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Therefore,  by  generating  nomenclature  to  more  accurately  represent  the  intended 
meanings  of  various  attributes,  and  collapsing  across  equivalent  conceptual  and 
operational  definitions,  a  more  concise,  attribute  compilation  was  generated.  (See 
Appendix  B.)  Additionally,  four  general  A/T  decision  attribute  categories  were 
identified  from  the  original  literature  review;  the  personnel,  task,  equipment,  and 
environment.  Therefore,  in  conjunction  with  the  compilation  process,  each  of  the 
attributes  were  appropriately  categorized. 

Based  on  this  work,  three  criteria  were  employed  in  the  selection  of  the  decision 
attributes  to  be  used  as  dependent  measures  in  this  study.  First,  the  attributes  and 
corresponding  definitions  were  to  be  taken  directly  from  the  compiled  list.  This  would 
minimize  inter-subject  variability  due  to  individual  interpretations.  Second,  the 
attributes  chosen  must  adequately  represent  a  cross-section  of  the  four  general 
categories.  And  third,  the  number  of  attributes  chosen  must  be  sufficiently  small  in 
order  to  be  statistically  manageable.  Based  on  these  criteria,  the  following  thirteen  A/T 
decision  attributes  were  selected  for  analysis  in  Study  1.  (See  Table  3.) 

Attribute  Relalionships 

As  indicated  earlier,  the  majority  of  previous  aiding  and  training  research  has 
focused  on  identifying  only  the  relationships  between  the  decision  attributes  and 
potential  A/T  solutions.  While  quantitatively  easy  to  analyze  and  interpret,  this 
approach  fails  to  address  the  more  subtle  complexities  of  these  decisions  in  a  realistic 
environment.  Understanding  the  influence  of  the  attributes  on  intermediate  decisions 
within  the  total  trade-off  process  is  equally  important  to  the  A/T  analyst  responsible  for 
performing  a  system  evaluation.  For  example,  "What  factors  (if  any)  prevent  the  use  of 
aiding?  or  training?",  and  "If  both  aiding  and  training  are  applicable  to  the  task;  are 
either  alone  sufficient?".  Study  1  addresses  these  concerns  through  a  series  of 
analyses  intended  to  identify;  1)  specific  barriers  to  aiding  and/or  training,  and  2.) 
attributes  indicating  aiding  or  training  sufficiency  for  that  task. 
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•  Aptitude 

•  Experience 

•  Number  of  Steps 

•  Mental  Skills 

•  Motor  Skills 

•  Frequency 

•  Performance  Rate 

•  Criticality 

•  Hazards 

•  Information  Amount 

•  Number  of  Components 

•  Training  Resources 

•  Aiding  Technologies 


Table  3.  List  of  Decision  Attributes 
Expert  Decision  Alternatives 

The  majority  of  previous  A/T  research,  with  few  exceptions  (Booher,  1978),  has 
been  additionally  constrained  by  the  limited  number  of  alternative  solutions 
addressed.  Allowing  only  the  choice  to  aid,  train,  both,  or  neither  eliminates  the 
potential  for  various  trade-offs  to  arise.  Such  a  situation  is  in  direct  conflict  with  the 
stated  goals  of  this  effort.  Therefore,  in  Study  1  the  analytical  investigation  of  the 
attribute-solution  relationships  was  extended  to  include  specific  A/T  approaches  and 
techniques.  (See  Table  4.) 

Note  that  the  goal  of  this  study  was  simply  to  identify  those  decision  attributes 
which  experts  consider  most  influential  in  their  aiding  and  training  trade-off  decisions. 
No  attempt  was  made  to  quantify  or  qualify  these  relationships.  The  intended  by¬ 
product  of  these  investigations  is  a  guide  to  more  detailed  research  focused  on 
determining  the  nature  of  these  relationships. 
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A  t..  Approaches 

Training  Approaches 

•  Display  Information 

•  Information  Presentation 

•  Monitor  Performance 

•  Modeling 

•  Assist  Execution 

•  Coaching 

•  Exploration 

Aiding  Techniques 

•  Information  Aid 

Training  Techniques 

•  Procedural  Aid 

•  Classroom 

•  Algorithmic  Aid 

•  Independent  Study 

•  Heuristic  Aid 

•  Simulators 

•  Assistive  Aid 

•  On-the-Job  Training 

Table  4.  List  of  Aiding/Training  Approaches  and  Techniques 


Subjects 

There  are  essentially  four  candidate  subject  populations  for  such  a  study:  1) 
operational/field  experts,  2)  A/T  developers,  3)  A/T  decision  makers,  and  4)  A/T 
researchers.  The  A/T  experiences  of  each  population  range  from  pragmatically 
oriented  applications  to  theoretically  oriented  research.  The  goal  of  this  effort  was  to 
focus  on  the  more  theoretic  orientation,  investigating  the  rules  and  heuristics 
employed  by  experts  in  making  sound  A/T  decisions.  The  target  population,  therefore, 
were  A/T  researchers  from  the  government,  academia,  or  industry  with  prior 
Department  of  Defense  (DoD)  related  A/T  experience. 

Seven  aiding  and/or  training  experts  participated  in  this  decision  policy  survey. 
Each  of  the  seven  subjects  had  a  minimum  of  4  years  prior  research  experience  in 
aiding  or  training  related  issues.  All  seven  were  civilian  contractors,  six  of  whom  had 
been  previously  involved  (in  a  contractor  or  government  employee  capacity)  with  A/T 
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programs  related  to  Department  of  Energy  (DoE),  Department  of  Defense  (DoD),  or 
NASA  systems. 

Participation  in  the  survey  was  strictly  voluntary  and  non-compensatory. 

Design 

In  order  to  empirically  evaluate  the  subjects'  A/T  decision  policies,  a  series  of 
hypothetical  scenarios  were  employed  to  provide  context.  Each  of  these  scenarios 
were  generated  based  on  select  Air  Force  Specialties  (AFSs),  Occupational  Survey 
Report  (OSR)  tasks,  and  the  13  decision  attribute  values.  Although  the  A/T  situations 
to  be  analyzed  by  the  experts  were  hypothetical  in  nature,  employing  actual  AFSs  and 
their  associated  OSR  tasks  served  two  important  roles.  First,  it  provided  the 
contextual  realism  necessary  for  the  subjects  to  relate  to  the  A/T  issue  addressed  in 
the  scenario.  And  second,  it  ensured  that  the  expert  decision  policies  captured  directly 
addressed  Air  Force  related  issues. 

Two  AFSs  chosen  for  this  analysis:  the  454X0,  Jet  Engine  Maintenance 
Technician,  and  the  811 XX,  Security  Police.  Their  selection  was,  to  a  great  extent, 
based  on  the  current  development  of  other  Air  Force  manpower  and  personnel 
modeling  efforts  (i.e..  Advanced  On-the-job  Training  System,  Training  Decisions 
Simulation,  etc.).  This  selection  strategy  supported  current  Air  Force  emphasis  on 
future  integration  of  proposed  manpower-personnel  models. 

Task  selection  was  based  on  an  initial  pool  of  48  tasks  (i.e.,  24  tasks  from  each 
AFS)  extracted  from  the  OSR  Occupational  Research  Data  Bank  (ORDB)  task  listing. 
These  tasks  were  selected  based  on  task  difficulty,  the  percentage  of  personnel 
performing  the  task,  and  on  the  implicit  goal  of  selecting  a  representational  cross- 
section  of  task  types  for  each  AFS.  A  3x3  criterion  table  was  developed  by  combining 
the  high,  medium,  and  low  levels  of  task  difficulty  and  percentage  of  personnel 
performing  the  task.  Each  level  of  task  difficulty  was  quantitatively  defined  in 
accordance  with  the  ORDB  description  (i.e.,  high  =  6.0  and  greater,  medium  =  4.0-5. 9, 
and  low  s  3.9  and  less).  The  corresponding  values  for  the  percentage  of  personnel 
performing  the  task  were  established  through  subjective  analysis  of  the  available 
ORDB  data  (i.e.,  high  =  50-100%,  medium  =  25-49%,  and  low  =10-24%).  Those  tasks 
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performed  by  less  than  10%  of  the  AFS  personnel  were  not  considered  as  candidates. 
This  decision  reflected  the  primary  concern  regarding  the  availability  of  Subject  Matter 
Expert  (SME)  data,  and  the  practicality  of  training  such  a  small  population  of  personnel 
(training  emphasis  is  highly  correlated  with  the  percentage  of  personnel  performing 
the  task).  Attention  was  also  given  to  selecting  tasks  which  appropriately  represented 
the  particular  AFS.  This  was  implemented  by  selecting  a  representative  sample  of 
administration,  supervision,  instruction,  documentation,  and  task  performance 
(cognitive  and  motor)  type  tasks. 

The  initial  list  of  48  tasks  was  then  further  reduced  to  the  final  24  based  on  task 
type;  Problem  Solving  versus  Non-Problem  Solving.  (Note:  The  decision  to  constrain 
the  evaluation  to  a  total  of  24  task  scenarios  was  based  on  expected  time  constraints.) 
"Problem-solving”,  in  this  study,  was  operationally  defined  as  a  task  characteristic 
which  requires  analytic  cognitive  processing  capabilities  (i.e.,  requires  some  form  of 
independent  evaluation).  In  contrast.  "Non-Problem  Solving”  tasks  require  more 
predetermined  procedurally  oriented  behaviors.  For  this  final  winnowing  process 
twelve  (12)  tasks  of  each  type  were  chosen;  six  per  AFS. 

Tables  5  and  6  list  the  final  24  tasks  utilized  in  this  survey. 

The  allowable  decision  attribute  values  were  operationally  defined  as;  high, 
medium,  low,  or  not  applicable.  This  scheme  provided  a  consistent  scale  across 
which  each  of  the  proposed  tasks  could  be  measured  -  independent  of  specific  units 
of  measure.  A  complete  matrix  representing  all  possible  value  combinations  of  the 
thirteen  decision  attributes  was  generated,  and  from  this  matrix  the  24  combinations 
for  analysis  were  selected.  Selection  of  the  final  24  attribute  value  combinations  was 
based  on  the  subjective  criteria  of  "reasonableness"  and  "appropriateness".  The 
"reasonableness"  criterion  rejected  those  combinations  which  were  intuitively 
unrealistic  (e.g.,  those  combinations  in  which  all  the  attributes  were  rated  high  or  low). 
The  "appropriateness"  criterion  was  directly  related  to  the  process  of  mapping  the 
attribute  value  combinations  to  one  of  the  selected  tasks.  More  specifically,  it 
addressed  the  appropriateness  of  the  combination  given  the  reality  of  the  task  (e.g.,  a 
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•  Adjust  Daily  Maintenance  Plans  to  Meet  Operational 
Commitments  (B28) 

•  Draft  Recommendations  for  Changes  in  Equipment  (B40) 

•  Identify  Recurring  Troubles  on  Equipment  by  Deficiency 
or  Service  Reports  (B60) 

•  Evaluate  Technical  Order  Improvement  Report  (C74) 

•  Counsel  Trainees  on  Training  Progress  (D97) 

•  isolate  Malfunction  w/  Engine  System  (G322) 

•  Annotate  or  Initiate  AFO  Form  98  {Jet  Engine  Afterburner  ♦ 
Inspectioi.  Record)  (E130) 

•  Operate  Computer  Automated  Maintenance/  Management 
Systems  (G350) 

•  Remove/Install  Safety  Devices  on  Engine  Component  (G433) 

•  Service  Starter  Units  (G470) 

•  Test  Gear-Box  Carbon  Seals  (G473) 

•  Inspect  Aircraft  Throttle  Controls  for  Freedom  of 
Movement  (1556) 


Table  5.  Jet  Engine  Maintenance  Technicians  (454X0)  Tasks 
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•  Counsel  Personnel  on  Personal  or  Military-related 
Problems  (B38) 

•  Plan  OJT  Training  Programs  (D156) 

•  Write  Security  Police  Computer  Programs  (E231) 

•  Evaluate  Situations  at  Incident  Scenes  (F275) 

•  Respond  to  Alarms  (1427) 

•  Perform  Hostage  Negotiations  (P578) 

•  Conduct  Inspection  of  Vehicles  at  installation  Entry  or 
Restricted  Area  Entry  Points  (F247) 

•  Employ  Life-Saving  Techniques  (F269) 

•  Plot  Accidents.  Incidents  on  Maps  (F304) 

•  Conduct  Anti-Robbery  Training  (H345) 

•  Assemble  or  Disassemble  M-16  Rifle  (0542) 

•  Employ  Individual  Night  Movement  Techniques  (0618) 


Table  6.  Security  Police  (81 1XX)  Tasks 


41 


combination  with  a  high  Hazard  value  would  be  inappropriate  for  a  "Complete  DO 
Form  1749"  task). 

Once  the  selection  and  mapping  processes  were  complete  a  hypothetical 
scenario  was  written  for  each  task  and  a  task  profile  generated.  The  context  of  each 
scenario  dealt  with  systems  and/or  situations  analogous  to  those  operationally 
existent,  but  was  kept  generic  to  avoid  potentially  confounding  biases  due  to  expert 
predispositions.  The  content  validity  of  each  scenario  was  then  evaluated  by  an  in- 
house  engineer  familiar  with  the  Jet  Engine  Technician  and  Security  Police  domains. 
The  task  profile  formed  a  graphical  representation  of  the  attribute  value  combination  of 
each  task  and  served  as  a  pictorial  summary  of  the  task  scenario.  (See  Appendix  F  for 
a  series  of  example  scenarios/profiles.) 

Study  1  employed  a  within-subjects  design;  each  subject  responding  to  an  eleven 
(11)  point  questionnaire  (See  Appendix  F)  for  each  of  the  24  hypothetical  A/T 
scenarios.  The  order  of  scenario  presentation  was  counter-balanced  by  AFS  (454X0  / 
81 XX)  and  task  type  (problem-solving  /  non-problem-solving).  The  order  of  the  6  tasks 
within  each  block  was  completely  randomized. 

The  questionnaire  was  divided  into  three  parts:  aiding  solution  specification, 
training  solution  specification,  and  combined  solution  specification. 

Three  issues  were  addressed  in  both  the  aiding  and  training  solution  specification 
parts;  barriers  to  and  sufficiency  of  the  respective  solution  type,  and  approach/ 
technique  preference.  For  the  two  former  issues,  subject  feedback  was  elicited  in  the 
form  of  a  nominal  exists  or  does  not  exist  response.  Approach/Technique  preference, 
in  contrast,  was  measured  as  the  percentage  of  hypothetical  resources  the  subject 
would  willingly  commit  to  employing  that  solution.  This  approach  enable  the  subjects 
to  employ  either  an  "all-or-none"  strategy  or  to  recommend  a  combination  of 
alternatives  with  a  relative  measure  of  merit. 

The  role  of  the  combined  solution  specification  part  of  the  questionnaire  was  to 
elicit  subject  preference  for  employing  the  A/T  approaches  and  techniques 
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recommended  earlier  in  the  questionnaire.  A  similar  resource  allocation  procedure 
was  employed  with  aiding  or  training  being  the  only  alternatives. 

Procedures 

Each  candidate  subject  was  contacted  and  given  a  brief  explanation  of  the  study 
and  the  estimated  time  requirement.  Those  experts  indicating  interest  and 
commitment  were  scheduled  to  participate,  at  their  convenience. 

At  the  beginning  of  each  scheduled  evaluation,  each  expert  was  briefed  on  the 
purpose  of  the  study,  its  role  in  the  JATAT  project,  and  the  expectations  of  their 
participation  (i.e.,  time,  procedures,  confidentiality).  Each  subject  was  supplied  with  a 
Task  Scenario  Notebook  (which  included  all  24  task  scenarios  and  profiles  in  proper 
presentation  order  and  a  copy  of  the  Decision  Attribute  Definition  Sheet)  and  a 
corresponding  package  of  Subject  Response  Sheets.  (See  Appendix  F.) 

No  time  constraints  (other  than  the  subject’s  available  time)  were  imposed  on  the 
task  performance. 

At  least  one  experimenter  was  available  during  the  entire  test  to  answer  any 
procedure-related  questions. 

During  the  post-test  debrief,  subjects  were  questioned  concerning;  1 .)  the  content 
validity  of  the  task  scenarios,  2.)  individual  decision  paradigms  employed,  and  3.)  the 
perceived  robustness  of  the  survey  procedure  for  A/T  decision  policy  capturing. 

The  questionnaire  responses  were  transformed  into  a  database  format  to  serve  as 
input  to  the  statistical  analyses. 

Analyses 

The  goal  of  the  analysis  was  three-fold:  identify  those  decision  attributes  which 
clearly  influence  expert  selections  of  A/T  techniques,  specify  linear  models  for  making 
A/T  decisions,  and  investigate  trends  in  data  for  areas  of  future  research.  In  order  to 
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address  each  of  these  goals,  three  independent  statistical  analyses  were  performed  -- 
linear  stepwise  regression  analyses,  discriminant  function  analyses,  and  summary 
statistics  analysis,  respectively. 

Each  analysis  was  performed  by  subject.  I'his  eliminated  confounding  results  due 
to  regression  toward  the  mean  of  expert  decision  approaches;  ensuring  that  individual 
decision  policies  were  independently  addressed,  intact. 

The  linear  stepwise  regression  analysis  of  the  13  decision  attributes  (predictors) 
and  recommended  A/T  approach  and  technique  percentages  (predicted  values)  was 
performed  using  a  default  alpha  of  0.150  to  enter  and  remove  attributes  from  the 
equation.  Figures  13*16  indicate  the  number  of  occurrences  of  each  attribute,  across 
experts,  for  a  given  approach  or  technique.  Appendix  D  presents  the  direction  and 
magnitude  of  the  coefficient  for  that  attribute. 

Figure  1 3  indicates  that  the  subjects  weighted  aptitude,  frequency,  and  training 
resources  most  heavily  in  applying  an  Information  Display  approach  to  the  task.  While 
the  number  of  components  (a  measure  of  task  complexity)  was  most  emphasized 
choice  for  Assisting  Execution. 

Criticality,  hazards,  and  the  information  amount  were  found  to  be  most  important  in 
the  selection  of  an  Information  Aid,  while  experience  and  training  resources  are  key  in 
the  selection  of  a  Procedural  Aid.  (See  Figure  14.)  Additionally,  aptitude,  frequency, 
criticality,  and  information  amount  are  the  most  heavily  weighted  attributes  in  the 
selection  of  a  Heuristic  Aid.  Figure  14  also  indicates  a  low  level  of  intra-subject 
consistency  in  their  applications  of  Algorithmic  and  Assistive  Aids. 

There  is  a  much  greater  consistency  among  the  subjects  in  their  attribute 
weighting  schemes  in  making  training  related  decisions,  as  seen  in  Figures  15  and  16. 
Most  notable  are  the  emphases  on  information  amount  for  selecting  the  Presentation 
of  Information,  and  performance  rate,  frequency,  and  aiding  technologies  on 
recommending  the  use  of  Exploration  as  training  approaches.  Motor  skills  and 
criticality  also  manifest  high  consensus  in  the  selection  of  a  Classroom  as  the  training 
technique  of  choice.  And  finally,  mental  skills  and  frequency  are  heavily  weighted  in 
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Display  Information 


Monitor  Performance 


Aptitude 
Experience 
Number  of  Steps 
Mental  Skills 
Motor  Skills 
Frequency 
Performance  Rate 
Criticality 
Hazards 

Information  Amount 
Number  of  Components 
Training  Resources 
Aiding  Technologies 


1  2  3  4  5 


Aptitude 
Experience 
Number  of  Steps 
Mental  Skills 
Motor  Skills 
Frsqusncy 
Psrfwmance  Rate 
Criticaiity 
Hazards 

Information  Amount 
Number  of  Components 
Training  Resources 
Aiding  Tschndogies 


Assist  Execution 


Aptitude 
Experience 
Number  of  Steps 
Mental  Skills 
Motor  Skills 
Frsqusncy 
Performartcs  Rate 
Criticality 
Hazards 

Information  Amount 
Number  of  Components 
Training  Resources 
Aiding  Technologies 


1  2  3  4  5 


Rgure  13.  Summary  of  Suh'sct  Recommendations  for  Aiding  Approaches 
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Information  Aid 


Procedural  Aid 


Aptitude 
Experience 
Number  of  Steps 
Merrtal  Skills 
Motor  Skills 
Frequency 
Performance  Rate 
Criticality 
Hazards 

Information  Amount 
Number  of  Components 
Training  Resources 
Aiding  TechrwIogies 


1  2  3  4  5 


Aptitude 
Experience 
Number  of  Steps 
Merrtal  Skills 
Motor  Skills 
Frequsrrcy 
Perforrrrarrce  Rats 
Criticality 
Hazards 

Infornration  Amount 
Number  of  Components 
Tr^ning  Resources 
Aiding  Tschndogies 


1  2  3  4  5 


Algorithmic  Aid 


Heuristic  Aid 


Aptituds 
Expsriencs 
Number  of  Steps 
MentsI  Skills 
Motor  Skills 
Frsqusncy 
Performance  Rate 
Criticality 
Hazards 

Information  Amount 
Number  of  Components 
Training  Resources 
Aiding  Technologies 

1  2  3  4  5 


Aptitude 
Experience 
Number  of  Steps 
MentsI  Skills 
Motor  Skills 
Frequency 
Performance  Rate 
Criticality 
Hazards 

Information  Amourrt 
Number  of  Components 
Training  Resources 
Aiding  Tschnoiogies 


1  2  3  4  5 


Assistive  Aid 


Aptitude 
Experience 
Number  of  Steps 
Mental  Skills 
Motor  Skills 
Frequency 
Performance  Rate 
Criticality 
Hazards 

Information  Amount 
Number  of  Components 
Training  Resources 
Aiding  Technologies 

1  2  3  4  5 


Figure  14.  Summary  of  Sut^ect  Recommendations  for  Aiding  Techniques 
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Information  Presentation 


Modeling 


Aptitude 
Experience 
Number  of  Steps 
Mental  Skills 
Motor  Skills 
Frequency 
Performance  Rate 
Criticality 
Hazards 

Information  Amount 
Number  of  Components 
Training  Resources 
Aiding  Technologies 


Aptitude 
Experience 
Number  of  Steps 
Mental  Skills 
Motor  Skills 
Frequency 
Performance  Rate 
Criticality 
Hazards 

Information  Amount 
Number  of  Components 
Training  Resources 
Aiding  Technologies 


Coaching 


Exploration 


Aptitude 
Experience 
Number  of  Steps 
Mental  Skills 
Motor  Skills 
Frequency 
Performance  Rate 
Criticality 
Hazards 

Irrformation  Amount 
Number  of  Comporwnts 
Training  Resources 
Aiding  Technologies 


1  2  3  4  5 


Aptitude 
Experience 
Number  of  Stepe 
Mental  Skins 
Motor  Skills 
Frequency 
Performance  Rate 
Criticality 
Hazards 

Information  Amount 
Number  of  Components 
Training  Resources 
Aiding  Technoiogiss 


Figure  15.  Summary  of  Sut^ect  Recommendations  for  Training  Approaches 
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Classroom 


Independent  Study 


Aptitude 
Experience 
Number  of  Stepe 
Mental  Skills 
Motor  Skills 
Frequency 
Performance  Rate 
Criticality 
Hazards 

Information  Amount 
Number  of  Components 
Training  Resources 
Aiding  Technologies 


Aptitude 
Experience 
Number  of  Stepe 
Mental  SkiUs 
Motor  Skills 
Frequency 
Performance  Rate 
Criticality 
Hazarde 

Information  Amount 
Number  of  Components 
Training  Resources 
Aiding  Technologies 

1  2  3  4  5 


Simulators 


On-the-Job  Training 


Aptitude 
Experience 
Number  of  Stepe 
Mental  Skills 
Motor  SkiUs 
Frequency 
Performance  Rate 
Criticality 
Hazards 

Information  Amount 
Number  of  Components 
Training  Resources 
Aiding  Technologies 


Aptitude 
Experience 
Number  of  Stepe 
MenM  SkiUs 
MotorSkiUs 
Frequency 
Performance  Rate 
Criticality 
Hazarde 

Information  Amount 
Number  of  Components 
Training  Resources 
Aiding  Technologies 


Figure  16.  Summary  of  Subject  Recommendations  for  Training  Techniques 
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the  decision  to  apply  Simulators  as  are  aptitude,  mental  skills,  frequency,  and  hazards 
in  recommending  On-the-job  Training. 

Linear  discriminant  function  analyses  were  used  to  specify  the  models  employed 
by  each  expert  in  answering  the  following  questions;  1.)  "in  what  conditions  did 
barriers  to  either  aiding  or  training  exist  ?",  2.)  "in  what  conditions  were  either  aiding  or 
training  alone  sufficient  to  perform  the  task  in  question?”  and,  3.)  "When  was  a 
combination  of  aiding  and  training  necessary?".  In  this  case,  the  13  decision  attribute 
values  again  served  as  predictors,  while  barriers,  sufficiency  and  combination  now 
served  as  the  predicted  values.  Figures  17-19  summarize,  across  subjects,  the 
number  of  occurrences  of  each  attribute  as  one  of  the  top  three  weighted  attributes  for 
that  model.  Appendix  E  contains  the  function  coefficients  for  each  expert  model. 

It  is  clear  from  Figure  17  that  aptitude  dominates  as  the  most  influential  attribute 
(followed  by  number  of  steps  and  mental  skills)  in  the  subjects'  determination  that 
aiding  was  not  applicable  to  a  specified  task.  Not  surprisingly,  aptitude  was  also 
influential  in  determining  if  aiding  alone  was  sufficient  for  a  specified  task.  (See  Figure 
18.)  This  was,  however,  moderated  by  consideration  of  performance  rate  and  the 
number  of  components. 

The  determination  of  training  sufficiency  has  a  much  greater  inter-  and  intra¬ 
subject  consistency  than  the  determination  of  Barriers  to  Training;  relative  to  the 
equivalent  comparison  for  aiding. 

Subject  selection  of  both  aiding  and  training  as  solution  components  placed  the 
majority  of  emphasis  on  those  attributes  related  to  the  cognitive  complexity  of  the  task 
(i.e.,  experience,  number  of  steps,  mental  skills,  and  number  of  components).  (See 
Figure  19.) 

Investigation  of  the  summary  statistics,  while  providing  less  detailed  information 
regarding  specific  decision  models,  provides  the  following  general  policy  trend 
information. 
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Aiding  Barriers 


Training  Barriers 


Aptitude 
Experience 
Number  of  Stepe 
Merrtal  Skills 
Motor  Skills 
Frequency 
Performance  Rate 
Criticality 
Hazards 

Information  Amount 
Number  of  Components 
Training  Resources 
Aiding  Technologies 


1  2  3  4  5 


Aptitude 
Experience 
Number  of  Steps 
Mental  Skills 
Motor  Skills 
Frequency 
Performance  Rate 
Criticality 
Hazards 

Information  Amount 
Number  of  Components 
Training  Resources 
Aiding  Technologies 


1  2  3  4  5 


Figure  1 7.  Summary  of  Subject  Recommendations  for  Aiding/T raining  Barriers 


Aiding  Sufficiency 


Training  Sufficiency 


Aptitude 
Experience 
Number  of  Stepe 
Mental  Skills 
Motor  Skills 
Frequency 
Performartce  Rate 
Criticality 
Hazards 

Information  Amount 
Number  of  Componertts 
Training  Resources 
Aiding  Technologies 


Aptitude 
Experience 
Number  of  Stepe 
Mental  Skills 
Motor  Skills 
Frequency 
Perfwmance  Rate 
Criticality 
Hazarda 

Information  Amount 
Number  of  Components 
Training  Resources 
Aiding  Technologies 


Figure  18.  Summary  of  Subject  Recommendations  for  Aiding/Training  Sufficiency 


Aiding/T  raining  Combination 


/Aptitude 
Experience 
Number  of  Steps 
Mental  Skills 
Motor  Skills 
Frequency 
Performance  Rate 
Criticality 
Hazards 

Information  Amount 
Number  of  Components 
Trainirtg  Resources 
Aiding  Technologias 


Figure  19.  Summary  of  Subject  Recommendations  for  Aiding/Training  Combinations 
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In  general,  the  more  complex  the  aiding  device  the  less  likely  subjects  were  to 
recommend  its  use.  (See  Figure  20.)  Two  factors  potentially  account  for  this  trend;  1 .) 
subjects  employed  a  "simplicity  first"  approach  (i.e.,  recommend  the  most  simple 
device  that  will  satisfy  the  requirements),  and/or  2.)  the  tasks  used  in  the  evaluation 
were  not  sufficiently  complex  to  warrant  more  sophisticated  aiding  devices.  One 
exception  to  this  trend  is  the  increased  number  of  recommendations  to  apply  Heuristic 
Aids  to  Problem  Solving  tasks.  This  distinction  is  further  accentuated  by  the 
significantly  lower  number  of  recommendations  for  Heuristic  Aids  applied  to  Non- 
Problem  Solving  tasks. 

A  similar  difference  exists  for  the  application  of  Procedural  Aids.  Subjects  were 
more  likely  to  recommend  Procedural  Aids  for  Non-Problem  Solving  tasks  than  for 
Problem  Solving  tasks. 

Figure  21  indicates  that  subjects  were  reluctant  to  recommend  any  type  of 
individual  training  program  (i.e..  Exploration  or  Independent  Study)  regardless  of  task 
type. 

Another  observation  from  Figure  21  is  that  Presentation  and  Modeling  are 
recommended  less  often  for  Problem  Solving  tasks  than  for  Non-Problem  Solving 
tasks.  The  rationale  for  this  decision  behavior  may  be  found  in  the  additional 
indication  that  Coaching  is  recommended  as  an  approach  to  training  Problem  Solving 
tasks  more  frequently  than  either  Presentation  or  Modeling.  It  appears  that  subjects 
tend  to  emphasize  the  need  for  individual  operator/maintainer  involvement  in  learning 
to  perform  in  dynamic  task  situations. 

A  review  of  the  summary  statistics  in  Tables  7-14  substantiate  the  earlier  decision 
to  analyze  each  subject's  decision  model  independently  due  to  expected  differences 
in  A/T  decision  approaches.  For  instance.  Subjects  2  and  4  were  much  more  training 
oriented;  recommending  far  fewer  aiding  applications  relative  to  other  subjects. 
Subject  2  was  particularly  unique  in  that  they  recommended  the  use  of  Exploration 
and  Independent  Study  significantly  more  than  any  other  subject.  While  Subjects  1 .3, 
and  5  relied  heavily  on  the  use  of  Information  Aids,  Classroom  instmction,  and  On-the- 
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App.  «1  App.  «2  App.  «3  Tech.  «1  Tech.  «2  Tech.  «3  Tech.  «4  Tech. 

Aiding  Approaches  and  Techniques 
«  Problem  Solving  a  Non-Problem  Solving 


Figure  20.  Summary  Statistics:  Subject  Recommendations  of  Aiding 
Approaches  and  Techniques 
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App.  #1  App.  #2  App.  #3  App.  #4  Tech.  #1  Tech.  #2  Tech.  #3  Tech.  #4 

Training  Approaches  and  Techniques 
«  Problem  Solvirtg  a  Non-Problem  Solving 


Figure  21 .  Summary  Statistics;  Subject  Recommendations  of  Training 
Approaches  and  Techniques 


Table  8.  Summary  Data  for  Aiding  Techniques  by  Subject  -  Problem  Solving  Tasks 
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Table  10.  Summary  Data  for  Training  Techniques  by  Subject  --  Problem  Solving  Tasks 


Table  12.  Summary  Data  for  Aiding  Techniques  by  Subject  -  Non-Problem  Solving  Tasks 


Training  Approaches 
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Table  14.  Summary  Data  for  Training  Techniques  by  Subject  --  Non-Problem  Solving  Tasks 


Job  Training,  Subjects  6  and  7  recommended  a  more  uniform  application  of  the 
various  A/T  fechniques. 

As  stated  earlier,  the  goal  of  these  analyses  was  to  identify  significant  trends  and 
relationships  in  the  A/T  decision  models  employed  by  the  7  participants.  The  pursuit  of 
quantifying  or  qualifying  these  observations  is  left  to  future  research  efforts. 

Post-Test  Debriefs 

Validity  of  Scenarios  -  During  the  post-test  debriefs,  all  subjects  iridicated  that  the 
task  scenarios  and  profiles  were  understandable  and  reasonable. 

Expert  Decision-Making  Paradigms  --  Differences  in  expert  decision-making 
approaches  were  elicited  through  informal  discussions  in  which  the  subjects 
attempted  to  verbally  recall  the  rationale  employed  in  making  several  of  the  evaluation 
decisions. 

The  responses  indicate  that  while  some  experts  employ  a  more  performance- 
based  approach  (addressing  constraints  only  after  all  the  solution  alternatives  have 
been  identified),  others  clearly  rely  more  heavily  on  identifying  the  subset  of 
reasonable  solutions  and  determining  which  of  those  satisfy  the  requirements  (a 
constraint-based  approach).  Experts  employing  a  performance-based  approach 
emphasized  the  influence  of  the  personnel,  task,  and  equipment  attributes.  In  contrast, 
those  experts  employing  a  constraint-based  approach  spoke  mostly  of  solution 
technologies  and  available  resources. 

There  was  considerable  intra-subject  variability  regarding  the  sufficiency  of  the 
information  available  to  make  the  decision.  For  some  tasks,  it  was  felt  that  certain 
attributes  were  not  necessary  to  make  an  A/T  decision  and  that  the  available 
information  was  more  than  sufficient.  However,  the  converse  was  equally  true.  It  is 
unclear  whether  this  variation  is  due  to  inconsistencies  in  the  rationale  used  by  the 
expert,  or  due  to  differences  among  the  tasks.  In  any  case,  it  is  clearly  better  to  err  on 
the  side  of  too  much  detail  that  too  little. 
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Rnhiistnpss  of  the  Analysis  -  Additional  expert  input,  spe-vned  by  these 
discussions,  indicated  that  a  linear  statistical  analysis  of  multiple  selection  responses 
is  too  constrained  as  an  approach  for  thoroughly  investigating  the  dynamic  A/T 
decision  environment. 

Surprisingly,  when  early  pilot-test  evaluations  included  questions  regarding 
intermediate  decision  constructs  (e.g.,  cognitive  resource  loading  and 
learning/performance  implications),  subjects  indicated  difficulty  responding  to  the 
questionnaire.  In  support  of  their  feedback,  they  cited  a  resistance  to  framing  their 
decision  processes  in  pre-specified  terminology.  (These  questions  were  subsequently 
removed  from  the  remaining  questionnaires.) 

However,  post-test  subjective  feedback  indicated  that  such  constructs  would  have 
been  useful  as  post-decision  analysis  measures.  In  other  words,  intermediate 
constructs  do  play  a  subtle  role  in  making  such  decisions  but  it  is  easier  to  investigate 
their  impact  after  the  decision  has  been  made. 

Study  2.  Informal  Investlaatlon  of  the  Content/Construct 
Validity  of  the  JATAT  Model 

As  earlier  stated,  one  objective  of  thic>  phase  of  the  JATAT  project  was  to  perform 
an  informal  investigation  of  the  internal  validity  of  the  hybrid  A/T  decision  model.  More 
specifically,  the  purpose  of  this  investigation  was  to  determine  if,  at  a  general  level,  the 
content  of  the  JATAT  model  covers  a  representative  sample  of  the  A/T  decision 
domain  (content  validity)  and  if  it  adequately  addresses  the  theoretical  constructs  of  an 
A/T  decision  support  system  (construct  validity). 

Several  candidate  approaches  exist  as  means  for  pursuing  this  investigation; 
predictive,  model-based  evaluations,  historical  analyses  of  previous 
successes/failures,  and  subjective  evaluations  by  A/T  experts.  Each  of  these  vary  in 
their  required  level  of  effort  and  their  empirical  rigor. 

The  approach  to  be  chosen  for  this  effort  was  subject  to  several  important 
constraints,  primarily;  limited  time  and  resources,  and  the  relative  immaturity  of  the 
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model.  Simply  stated,  the  time  and  resources  required  to  perform  a  predictive,  model- 
based  approach  were  well  beyond  the  scope  of  this  effort.  This  method  would  entail 
conducting  an  exhaustive  search  of  current  Air  Force  systems  to  identify  a  potential  A/T 
deficiency,  generating  a  recommended  solution  with  the  JATAT  model,  and 
authorizing,  implementing  and  evaluating  the  resultant  performance  of  that 
recommendation.  A  historical  analysis  of  previous  aiding  and  training 
successes/failures  would  have  suffered  from  the  same  time  and  resource  limitation. 
Additionally,  it  would  potentially  lack  the  necessary  decision  attributes  values  and 
adequate  performance  measures.  In  either  case,  the  current  JATAT  model,  while 
structurally  complete,  lacks  the  detail  (i.e.,  quantitative/qualitative  relationships  among 
intermediate  constructs)  necessary  to  independently  arrive  at  specific  answers  to 
specific  problems. 

Subjective  expert  evaluation  through  informal  interviews,  in  contrast,  provides  an 
expeditious  method  to  investigate,  in  a  general  sense,  the  expected  inputs  and  outputs 
of  the  JATAT  model  against  previous  experience.  This  approach,  in  particular, 
enables  subjective  validation  of  what  does  exist  as  well  as  feedback  regarding 
perceived  deficiencies  in  the  current  model. 

Therefore,  Study  2  was  performed  using  a  series  of  informal,  semi-structured 
interviews  with  available  A/T  experts. 

Subjects 

Eight  experts  from  academia,  industry  and  the  DoD  participated  in  this  portion  of 
the  study.  None  of  the  subjects  participating  in  Study  2  had  participated  in  the  earlier 
statistical  survey.  Again,  all  subjects  had  prior  experience  in  government-related 
aiding  and/or  training  projects  and  all  subject  participation  was  voluntary  and  without 
compensation. 
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Procedures 


Each  candidate  subject  was  first  contacted  through  a  preliminary  screening  call. 
During  this  call,  the  purpose  of  the  interview  was  briefly  described  and  the  expert's 
level  of  interest  for  acting  as  a  participant  was  elicited. 

A  follow-up  contact  was  later  made  with  each  interested  candidate.  The  purpose 
of  this  call  was  to  address,  in  more  detail,  the  goals  of  the  interview  process,  expected 
time  requirements  and  scheduling  constraints.  Since  each  subject  was  participating 
on  a  volunteer,  non-compensatory  basis,  the  interview  sessions  were  limited  to  one 
hour  in  length.  Those  candidates  who  displayed  continued  interest  and  availability 
were  scheduled  for  an  interview  at  a  time  convenient  to  both  them  and  the 
experimenters. 

Each  scheduled  expert  was  sent  an  interview  packet  which  contained  an 
introductory  letter,  a  list  of  decision  attribute  definitions,  four  task  scenario/profile 
combinations,  and  a  summary  page  of  A/T  decision  questions.  The  task  scenario/ 
profiles  and  attribute  definitions  were  taken  directly  from  the  task  scenario  notebook 
used  in  Study  1.  (See  Appendix  F.)  The  A/T  decision  questions  were  compiled  from 
the  questionnaires  also  employed  in  Study  1.  The  purpose  of  this  packet  was  to 
familiarize  each  expert  with  the  scope  and  focus  of  the  current  JATAT  effort  and  the 
expectations  of  them  as  participants,  prior  to  the  interview. 

Each  interview  followed  a  pre-determined  protocol.  This  included  an  introduction 
to  JATAT  and  to  the  expert,  a  review  of  current  A/T  decision-making  and  related 
issues,  discussions  of  alternative  approaches,  and  a  critical  evaluation  of  the  current 
JATAT  model. 

During  the  introduction,  the  experts  were  given  a  brief  overview  of  the  JATAT 
project;  the  sponsor,  the  history,  the  goals,  and,  again,  the  objectives  of  the  interview 
session. 

Each  expert,  in  turn,  gave  a  short  dissertation  of  their  experiences  and  approach  to 
aiding  or  training  decision-making.  This  review  of  each  experts'  recent  experiences 


61 


with  aiding  and/or  training  decision  efforts  played  two  important  roles.  One,  it  provided 
the  experimenters  with  exemplary  detail  allowing  them  to  more  clearly  understand  the 
experts'  decision-making  approach.  And  two,  it  aided  the  experts  in  becoming  more 
actively  involved  with  the  interview  atmosphere.  (Note;  the  task  scenarios  included  in 
the  familiarization  packet  were  to  be  used  as  mental  triggers,  or  sources  of  discussion, 
if  necessary,  during  this  time.) 

The  interviewers  then  promoted  more  in  depth  discussions  of  the  general 
constructs  and  approaches  employed  by  the  expert  in  their  example.  This  led  to  further 
discussions  of  the  limitations  of  current  approaches  and  identification  of  viable 
alternatives. 

Finally,  the  current  JATAT  decision  aiding  model  was  presented  allowing  the 
expert  to  critically  evaluate  its  strengths  and  weaknesses. 

Summary  of  Responses 

In  review,  the  purpose  of  the  interview  activities  was  to  elicit  expert  opinion 
regarding  perceived  content/construct  validity  of  the  current  JATAT  decision  aiding 
model.  Therefore,  in  order  to  most  thoroughly  present  the  expert  responses,  this 
section  has  been  divided  into  three  major  portions,  each  of  which  presents  a 
compilation  of  the  issues  addressed  by  the  interviewees.  The  first  portion  presents  two 
strategic  issues  associated  with  future  JATAT  development.  The  second,  presents 
feedback  supporting  the  current  model.  And  the  third,  presents  feedback  identifying  its 
deficiencies. 

Strategic  Issues 

Even  from  among  the  wide  variety  of  expert  aiding  and  training  backgrounds,  two 
issues  were  nearly  unanimously  addressed  and  nearly  complete  consensus  was 
attained. 
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First,  the  development  of  a  decision  support  system  with  goals  and  objectives 
similar  to  the  current  JATAT  effort  would  be  of  great  value  to  the  aiding  and  training 
decision-making  communities.  Pursuit  of  such  a  system  is  imperative. 

And  second,  a  clear  and  distinct  identification  of  the  intended  user  population  is 
essential  to  the  development  of  such  a  system.  As  with  any  new  system,  an  ill-defined 
or  imprecise  definition  of  the  user  community  and  its  needs  endangers  the  potential 
success  of  such  a  system  at  its  very  out-set.  Therefore,  in  situations  in  which  no 
distinct  user  populace  currently  exists,  as  is  the  case  with  JATAT,  it  is  imperative  that 
the  support  requirements  of  related  user  communities  be  identified  and  addressed  by 
the  proposed  system.  Several  experts  recommended  that  Air  Force  Special  Programs 
Office  (SPO)  personnel.  Army  Training-Doctrine  (TRADOC)  Command, 
Combat/Training  Development  School  personnel.  Human  Resources  department 
personnel  and  their  military  and  civilian  counterparts  potentially  represent  such  user 
populations. 

Support  for  ttiflLCurrent  Model 

The  Use  of  Task  Analyses  -  During  the  discussions  of  current  A/T  decision 
processes,  experts  indicated  that,  too  frequently,  A/T  decision  solutions  are  reactive  in 
nature  and  driven  by  technology.  More  specifically,  customers  often  bias  an  analyst's 
investigation  toward  a  particular  solution  without  thoroughly  understanding  the 
requirements  of  the  problem.  A  typical  question  is,  ”Can  we  use  this  (some  specific) 
technology?"  Unfortunately,  the  evaluation  of  the  recommended  solution  is  often 
equally  reactive;  following  the  same  constraint-based  pattern.  ("Why  didn't  we  use  this 
technology?") 

A  constraint  driven  approach,  as  such,  is  highly  susceptible  to  fluctuations  in 
external  factors  (i.e.,  technology,  leadership,  organizational  policy,  etc.).  Therefore,  in 
order  to  assist  an  analyst  in  taking  a  more  proactive  approach  to  generating  and 
evaluating  recommended  A/T  alternatives  a  decision  support  system  must  be 
requirements  driven. 
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As  in  JATAT,  this  approach  requires  a  thorough  task  analysis  upon  which 
alternative  recommendations  are  based.  Once  a  viable  subset  of  requirements  based 
solutions  are  formulated,  final  recommendations  can  then  be  altered  in  response  to 
fluctuations  in  external  constraints.  In  this  way,  external  constraints  will  exhibit  much 
less  impact  on  the  overall  decision  process. 

The  Use  of  Intermediate  Explanations  -  In  order  for  a  decision  support/advisory 
system  to  be  useful,  the  user  must  have  enough  confidence  in  the  system's 
recommendations  to  use  it. 

One  important  factor  contributing  to  human-machine  confidence  is  an 
understanding  of  the  decision  model  employed  by  the  system.  In  situations  in  which 
the  user's  solution  and  the  system's  recommendation  conflicts,  the  user  must  assume 
the  role  of  decision  arbitrator.  In  this  capacity,  having  access  to  the  intermediate 
decisions  arrived  at  by  the  system  enables  the  user  to  modify  his  interpretation  and 
accept  the  system's  recommendation,  or  reject  the  system's  recommendation  in  favor 
of  his  original  solution.  In  the  absence  of  such  intermediate  explanations,  the  latter 
decision  is  simply  made  most  frequently.  In  other  words,  users  have  inherently  low 
confidence)  in  systems  which  accept  a  problem  definition  as  the  input  and  provide  only 
a  solution  as  the  output.  Therefore,  the  most  important  component  of  an  advisory 
system  is  the  critic  -  that  which  provides  intermediate  explanations  throughout  the 
decision  process. 

By  including  intermediate  decision  constructs  as  integral  components  of  the 
JATAT  model,  a  subsequent  decision  support  system  would  be  capable  of  providing 
the  interiediate  explanations  on  inquiry. 

The  Use  of  Simulation-Based  Analyses  --  Several  experts  suggest  that  in  order  for 
a  recommendation  to  be  considered  reliable,  it  must  be  accompanied  by  supporting 
performance  data.  While  the  scope  of  this  phase  of  the  JATAT  effort  has  focused 
solely  on  the  model  for  formulating  aiding  and  training  alternatives,  it  is  clear  that  the 
simulation-based  evaluation  component  of  the  Phase  2  hybrid  model  has 
considerable  merit. 
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luiMi.  ui  ina  \^uirBrn  iviuuei 

There  were  essentially  two  deficiencies  with  the  current  JATAT  model  identified  by 
the  interviewed  experts.  First,  the  attribute  categories  addressed  (personnel,  task, 
equipment,  and  environment)  in  the  model  do  not  reflect  organizational  influences. 
Although  difficult  to  quantify,  and  sometimes  more  difficult  to  define,  organizational 
issues  have  a  significant  impact  on  many  A/T  decisions.  And  second,  the  personnel 
attributes  analyzed  (aptitude,  experience)  do  not  address  specific  capabilities  and 
experiences  of  the  individual.  While  this  issue  is  addressed  at  the  task  level  in  the 
process  support  structure  ("Are  the  activities  trained  or  aided  in  another  AFS?")  a  more 
direct  approach  to  capabilities  with  respect  to  A/T  technologies  is  certainly  appropriate 
(i.e.,  "Have  the  personnel  had  any  experience  with  computer  based  training?"). 

SUMMARY 

The  objective  of  this  phase  of  effort  has  been  to  further  investigate  the  task 
analysis  and  trade-off  formulation  components  of  the  original  JATAT  methodology.  An 
integral  component  in  this  effort  was  to  be  the  development  of  a  conceptual  model  for 
selecting  alternative  A/T  techniques  for  a  given  task.  In  support  of  this  development, 
two  independent  evaluations  were  to  occur;  the  first,  a  statistical  analysis  of  the 
decision  attributes  which  contribute  to  the  A/T  decision  process,  and  the  second,  an 
informal  investigation  of  the  internal  validity  of  the  newly  formulated  model. 

During  the  past  year,  these  efforts  have  achieved  several  important  milestones 
toward  the  eventual  implementation  of  a  complete  JATAT  Decision  Support  System. 
First,  the  JATAT  General  Decision  Model  was  formulated.  This  model  responded  to 
the  deficiencies  of  earlier  approaches  by:  1.)  addressing  a  larger  array  of  aiding  and 
training  alternatives,  2.)  investigating  a  new  complement  of  attribute-solution 
relationships  through  the  inclusion  of  important  intermediate  decision  constructs,  and 
3.)  addressing  specific  operationally  oriented  inputs.  Pursuit  of  this  latter  component 
resulted  in  the  development  of  the  Process  Support  Structure  which  provides  the 
operational  context  necessary  to  the  General  Decision  Model  based  on  currently 
available  Air  Force  data  structures  (i.e.,  ORDB,  OSR  Task  Analyses,  etc.).  A  statistical 
survey  of  7  A/T  experts'  decision  policies  resulted  in  a  series  of  initial  observations 
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regarding  specific  attribute-solution  relationships  which  will  provide  guidance  for 
future  research  efforts  directed  at  quantifying/qualifying  those  relationships.  And 
finally,  a  series  of  semi-structured  interviews  with  other  A/T  experts  to  informally 
validate  the  structure  and  methodology  of  the  JATAT  model  provided  strong,  positive 
support  for  the  current  direction  of  the  JATAT  program. 
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APPENDIX  A 


Compilation  of  Job  Aiding/  Training  Attributes. 

•  A  literature  review  of  the  attributes  used  in  job  aiding/training  trade-off  decisions  was 
conducted  which  resulted  in  the  following  compilation. 

•  Only  the  enumerated  attributes  within  each  article  were  included  in  the  compilation. 

•  The  following  bullets  identify  some  of  our  findings  based  on  this  review  of  the 
literature: 

••  There  are  many  attributes  which  have  been  addressed  in  the  literature, 
some  of  which  have  multiple  labels  and/or  multiple  definitions. 

••  Some  attributes  are  defined  without  specific  recommendations,  while 
some  are  never  clearly  defined  at  all. 

••  Other  authors  are  content  to  make  recommendations  without  clearly 
defining  the  attributes. 

••  Very  little  of  the  literature  presents  specific  equations  or  algorithms  for 
relating  the  many  attributes. 


•  It  is  clear,  from  this  compilation,  that  there  is  currently  no  comprehensive  research 
which  accounts  for  all  of  the  attributes. 
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Author(s)  Variable  Level(s)  Definition  Recommendations 
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Author(s}  Variable  Level(s)  Definition  Recommendations 
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frequent  perfonnance 


71 


doesnl  exist  identify  tasks  that,  if  improperly 

performed,  can  cause  injury  or 
death  to  the  performer  or 

_  _  others.  _ 
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performance  aids,  refresher 
training,  or  both  3  months  and 
6  months  after  learning. 
Refresher  training  is  required 
1  year  later 


Author(s)  Variable  Level(s)  Definition  Recommendations 
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Author(s)  Variable  Level(s)  Definition  Recommendations 
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Author(s)  Variable  Level(s)  Definition  Recommendations 
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attribute  muitistep  or  multi-attribute 

_ situation  then  consider  aiding. 

error  consequence  high  If  the  consequences  of  error 

_ are  high  then  consider  aiding. 


APPENDIX  B 


Standardization  of  the  Job  Aiding/Training  Attributes. 

•  This  appendix  presents  the  culmination  of  an  attribute  standardization  effort  designed 
to  eliminate  redundancy  and  ambiguity  amongst  the  attributes  extracted  from  the 
literature  review. 

Common  names  for  the  attributes  have  been  provided  along  with  the  original 
variable  name.  For  example,  "technician  aptitude”  has  been  replaced  with  the  more 
generic  name  "personnel  aptitude." 

Where  multiple  authors  provided  similar  but  slightly  different  definitions  of  the  same 
attribute,  all  definitions  have  been  included. 
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lests. 
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New  Original  Definition  Authors 

Nomenclature  Variable  Name 
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New  Original  Definition  Authors 

Nomenclature  Variable  Name 
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New  Original  Definition  Authors 


APPENDIX  C 


Example  JATAT  Analysis 


87 


Introduction 


•  This  appendix  presents  an  example  aiding/training  decision  analysis  employing  both 
the  currently  available  approaches  reviewed  earlier  in  this  report  and  the  JATAT 
decision  process  and  its  supporting  decision  model. 

•  These  independent  analyses  enable  a  comparative  evaluation  of  the  competing 
approaches,  pin-pointing  the  strengths  and  weaknesses  of  each. 

Scenario 

•  A  new  electronic  measuring  device  for  troubleshooting  engine  system  malfunctions 
has  been  recently  developed  and  is  expected  to  be  operationally  implemented  in  the 
near  future. 

•  In  conjunction  with  implementing  the  new  device,  it  is  deemed  necessary  to  perform 
an  analysis  on  the  current  training/aiding  program  (making  modifications  where 
necessary)  to  ensure  a  trouble-free  transition  into  the  operational  setting. 

•  The  following  text  summarizes  the  scenario  in  the  context  of  the  454X0  Task  Analysis 
document. 

•  TASK  CHANGE:  One  task  which  is  influenced  by  the  new  measuring  device  is 
"troubleshoot  engine  system  malfunctions  on  flightline." 

•  ACTIVITIES  CHANGE:  None.  The  activities  of  this  task  are  the  same  regardless  of 
new  measuring  devices.  (The  activities  are  formulate  possible  causes  of  malfunction, 
remove/install  suspect  component,  inspect,  test/replace  suspect  component, 
functionally  test  system.) 

•  SKILLS  CHANGE:  Only  one  skill-  "use  measuring  devices"  is  influenced  by  the  new 
device. 
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•  KNOWLEDGE  CHANGE:  Two  types  of  knowledge  are  influenced  by  the  new 
device-  "interpret  gauges  and  meters",  and  "locate  parts/components/sections." 

•  Note  that  this  summary  corresponds  to  the  task  decomposition  stage  of  the  decision 
process  diagram  in  Figure  3. 

•  The  result  of  the  decision  flowchart  is  a  need  to  evaluate  the  new  knowledge  and 
skills. 


Analysis  Using  Currently  Available  Models 
FACTOR  IMPLICATION 


Harless,  Lineberry,  and  Rossett  &  Downes  Model  (See  Figure  22.) 


Task  definition:  clearly  defined 
Information  quantity:  low 
Training  resources:  few 
Performance  frequency:  infrequent 
Task  changeability:  infrequent 
Error  consequence:  high 


Aiding  >  Training 
Training  >  Aiding 
Aiding  >  Training 
Aiding  >  Training 
Training  >  Aiding 
Aiding  >  Training 


•  The  ">"  indicates  that  one  approach  is  recommended  over  the  other. 


"Barriers  to  Aiding"  Model  (See  Figure  23.) 

Time  required:  .5  hour 
Time  available:  1.5  hour 
Performance  rate:  3 

Conclusion:  Employ  Aiding 
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Goins  Model  (See  Figure  24.) 


Equipment  hazards:  no 
Task  criticality:  high 
Personal  hazards:  no 
New  skills  required:  no 
Performance  Frequency:  <  3  months 
New  knowledge  required:  yes 

Suggest:  Training  >  Aiding 


Conclusions 

•  From  the  Harless,  Lineberry,  and  Rossett  &  Downes  Model:  AID. 

(Of  the  first  six  variables,  four  suggest  aiding  while  two  suggest  training.  It  should  be 
noted  that  the  two  variables  which  point  to  training  are  logical  extrapolations  and  are 
not  mentioned  in  the  literature.) 

•  From  the  Barriers  to  Aiding  Model:  AID. 

(There  are  no  barriers  to  aiding  with  respect  to  time.) 

•  From  the  Goins  Model:  TRAIN. 

(Although  the  task  is  critical,  the  frequency  of  performance  is  less  than  three  months.) 

•  This  analysis  indicates  that  simple  equal-weighted  analysis  schemes  can  sometimes 
end  in  contradictory  results.  Unfortunately,  these  methods  do  not  provide  the  analyst 
wit^  more  specific  information  about  the  decision  process  and.  hence,  there  are  no 
means  of  resolving  the  conflict. 
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-  The  darkened  lines  indicate  the  path  taken 
through  the  network  in  this  example. 


•  A  simple  knowledge-based  approach,  in  contrast,  would  be  able  to  isolate  sp'.  tic 
inflection  points  (single  variables  which  contradict  the  conclusion)  providing  a  much 
clearer  rationale  behind  the  recommendations  of  the  analysis. 


Recommendations 

•  What  follows  is  a  hypothetical  example  of  the  results  of  a  simple  knowledge-based 
analysis. 

•  For  any  analysis,  there  will  be  certain  critical  attributes  which  have  a  particularly 
strong  influence  on  the  resultant  conclusion.  These  critical  attributes  can  either  be 
negatively  associated  with  the  outcome  (i.e..  barriers  to  aiding  or  training)  or  positively 
associated  with  the  outcome  (i.e..  critical  juncture  in  a  relationship  network). 

•  Identification  of  these  critical  attributes  is  important  to  forming  useful  explanations  of 
the  recommendations. 

•  It  seems  that  the  critical  attributes  In  this  tradeoff  (In  this  case,  favoring  aiding)  are; 
training  resources,  performance  frequency,  eiror  consequence,  and  task  criticality. 

•  IF  an  aid  can  be  inexpensively  produced,  and  the  performance  frequency  does  not 
demand  training,  THEN  AID. 

•  IF  the  training  for  the  now  measurement  device  is  less  expensive  than  the 
development  of  an  aid,  or  if  the  information  to  be  trained  is  not  extensive,  or  the 
cognitive  skills  are  not  demanding,  THEN  TRAIN. 

•  It  is  apparent  that  these  recommendations  are  not  particularly  insightful,  however 
they  do  point  out  potential  trade-offs  v;hich  the  JATAT  user  then  could  consider  in 
more  detail. 
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JATAT  Model  Analysis 


•  Given  that  the  applicable  knowledge  and  skills  for  the  analysis  have  been  identified 
(see  previous  section),  the  analysis  begins  with  Figure  4.  at  the  point  of  evaluating 
these  knowledge  and  skills. 

•  The  physical  requirement  of  the  "use  measuring  devices"  skill  is  essentially  a  manual 
dexterity  manipulation.  This  can  be  easily  accomplished  by  current  personnel. 
Therefore,  the  behavior  must  be  instructed. 

•  The  knowledge  component  of  this  skill  corresponds  to  the  "interpret  gauges  and 
meters"  knowledge  previously  identified  and  would  be  analyzed  accordingly. 

•  ft  appears  that  there  are  no  foreseen  barriers  to  aiding  for  either  of  these  knowledge 
types. 

•  The  high  task  criticality  and  the  new  knowledge  required  imply  that  aiding  of  this  task 
alone  would  be  insufficient. 

•  There  also  appears  to  be  no  foreseen  barriers  to  training.  Therefore,  some 
combination  of  training  and  aiding  are  appropriate  for  this  task.  (See  Rgure  25.) 
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Figure  25.  Exampte  JATAT  Path  Analysis 


APPENDIX  D 


Regression  Coefficients  for  Attribute-Solution  Relationships 


97 


Regression  Coefficients  for  Assist  Execution 
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Hazards  -14.868  8.885 

Information  Amount  12.601  -5 

Number  of  Components 

Training  Resources  -85 

Aiding  Technologies 
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Regression  Coefficients  for  Heuristic  Aid 
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Hazards 

Information  Amount 
Number  of  Components 
Training  Resources 
Aiding  Technologies 


Regression  Coefficients  for  Information  Presentation 
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Hazards 

Information  Amount  -14.691  -5.192 

Number  of  Components 
Training  Resources 
Aiding  Technologies 


Regression  Coefficients  for  Ciassroom 
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Regression  Coefficients  for  Simuiators 
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Regression  Coefficients  for  Expert 
Aiding/Training  Barrier,  Sufficiency,  and  Combination  Models 


106 


u 

09 

!5 

(D 

tj 

« 

!o 

a 

in 

ts 

0) 

in 

</3 


tj 

a 

in 

n 

w 

I 

in 


wl 

CM 

S 

in 

n 

c/] 


in 

(/: 


3 


_00  —  <0'^u.,<0^O^00C\J_<0 

ff-u-Soir^S^ooScooSiw 

[JJogcaevj^cvjo--. 

•0^00^000^00^0 
▼~iOi  lO)  I  lOi  lOi 


^  ^  ^  ^  O  ^  CO  ^  ^  ^  ^  Q  Q  ^ 

'O^OOOOOO^^O^^ 
^lOi  I  I  <  •  lOOlOO 


SoO'^l^CMCMrr^I’O^^ 


%  y  ^  w# 


CNJ 


® 

04 '  < ^  o  ^ 
^OOO—OO^O^OO^X 
O  I  I  I  O  •  )  O  •  O  I  I  o  o 


o  ^ 


S®i®S®3c4«-®JQgo»-r^ 

®  —  ^0»-^qqqCMO^qq 


^  9 


“^0003 


00 . _  - 

X<-{0^000^^000 
WWiWi  •  iWOi  I  I 


Q)  PD  1^  O) 


i  ssssssiSsssaJs 

9  ’T o  ^  9  ^  o  »- o  ^  ^  04 


r*  ®  52  -^r  h? 


0  -»-»--  -0»-0  .t-C4 

iiOiOOiiOOOlOO 


®^®^o)^®3®o>®r^or^ 

CM^O^O^.  0l]0000«»- 
OiOiOiOVoOOOOO 


w«^S€'i<'*oWr^C:r^c4®'^2 

oj^^^oHoo^-r-o^r-H 

d99do9od9oddd9 


c  o  S’ -2 .2  2  c  P  S  ^ 
^^u.a.oi£zi-< 


2 

o 

‘E 

s 


s 

< 

o 

v> 

(0 

c 

< 


CA 

c 

o 

O 

jg 

« 

o 

O 

c 

o 

‘w 

<A 

S 

? 

CE 


107 


108 


(O 

tj 

<D 

Z3 

<531 


TS 

03 

jQ 


031 


U?  «  «  ^  00  C\i 


3 


»  —  00  ^  <o 

2  r-  2  CO  U5 
tP  O  ^  CNJ  CvJ 


^CVJ.^.  .OCOCO  -  . 

9d9d99ddd9d999 


SS§«gu555C:®g‘e55^ 

^^0^0^0^000000 
O  O  I  w  I  O  »  O  ••till 


^  m ’“  —  <*>  rt»  .A  o  lA  0»  o*  <p  <*> 
®  ^^lA^CD5fzrlftK?f^OO^(0 

i  g"p5“S2"S°°92o 


OwiWiwOlOl  I  I  •  I 


tj 

a 

2 

a 

<531 

coi 


S;2SS§g§8 


^  SJ »-  »- 


O  A  <0 
I^o^og 
O  T-  ^  o  ° 


CN< 


O^OOO^aaOwwOwO 
-OiiiOOOtOOiOi 


»-o»S<m2-^^cocoS15S*^o 

C0C0”<O^<O00CM<O^A”iAm 
^Ol^CM^.  CVJOCMOl^SIJCM^ 
OOlOiOOOOiiiO* 


2 

3 

<n 


1  sSl5®^i®§«g^o°S 

fl  CM  2!  o  [z o  z  o  s  o  ir  •r- s 


% 

2 

3 

03 


c>!;zT-q^pj5c>^^ 
o^d^dd^d^dd^o^ 

lOiOi  lOtOi  lOiO 


ggoeoo^q^ooScM'M’n 

'i’Too^^o^^-o^cmoco 

99ddd9d9dd9ddd 


0) 

;g 

3 

03 

G> 

C 

"6 

o 

(0 

(0 


a 


p  Q.  X  ^  ^  S  «  c  (0 


0<11JZ 


3 

(0 

c 

o 

O 

se 

a> 

o 

O 

c 

o 

s 

s 

8* 


109 


c/31 


cm 


c/31 


<fl( 


€ 


^  m  Q 


(O 


00  Si  lO 


00  ^  lO 


to 


U) 


to  in  CM  T-  ^ 


iS  g oo  tt 

fvi  Q  <M  rt  T” 


CM 


9696ci999699oo 


g  CM  g  ^  q  q  S;  «  5;  -  «  -  CM 

cM9c>9999c>dd9c>99 


oSSocMw^cjcM^to^oo^l^ 

o 


'M-cooS^ldcM««^j;j«g«q 


999d99ddddddd® 


_^OQ_tOO  — 

®SSo”Sc'/o<®cM'-'‘" 

^  ^  CM  ^ 

00  •  ^3  •  •  ^ 

'0^00^000^0000 
'r~iOi  lOi  ■  tOi  I  I  I 


f^noicor- 
O  0>  CM  0> 

o  o 


fooJR^oofesS^'XfSoj^S 

'OO^^O^OOaOaaO 

T-ii00<0.i0»00i 


^^S®w^*®eo^2S9§ 

^ooq^oo^o^oOj^o 


5  S’-.59::S2«^55':^.99 


CM  ^  lO  to 


d9d9ddd9dd9999 


CO 


2? 

c 

9 

o 

ie 

o 

c 

c 


o 

CO 

SS 

(0 

c 

< 


M 

c 

o 

O 

se 

o 

o 

O 

c 

o 

'to 

CO 

£ 

cr 


110 


y  .  O  T-  .  o  o  ■  ■  ■  .  O  ■r-  o  o 


<oi 


V 

a 

2 

3 

(/) 


JO 

3 

C/3 


C/31 

«T| 


CM 

2 


£ 

3 

Cfl 


I 


^dd^dd^^^^oocbo 


®<  c'l  S  2  ^  00  S  ®  00  S  m 

^  in  o)  CO  CM  ®  in  S  s  w  5  ®  o  2 

cooeM»-.o.’-?o..o.^ 

dddd9d99d9‘?d99 


op  ^ 


on  cn o> , 


SE  ’-S®coO‘2ca®inin®i-^® 

5  ?o::«at5~sa!egqg« 


o9d9dd9d99d9oo 


.1  s 


SDrsr.  T-oo®--©!::: 


CO  o> 


Oi 


00 


O  O  O  CM 

d  d  d  d 


(A  CO  <o 
in 


CO  o  »- 

d  d  d 


r<oooof^S^i»c»ii!S«o<oco 


0^00 

d  d  d  d 


o  o 


o  o 
d  d 


o  o 


o  o  o 


OOOCM-.  —  _ 

dddd9dd9d99dd® 


^  ^  •■I.  w «r»  ® '<*■  ® 

ddd99d9d99d999 


w 

c 

9 

11 

Is 


w 

10  9 
I  9 

=  O 


(O 


(0 

c 

2 

E 

o 

O 

o» 

c 

c 

c3 


^  o>  oj  o»  g  o  o  5  r>*  So 2  ^ 

«o  ^  00  CO  §  o  CO  <5  CO  S  2  h»  <o  S  g» 

O  O  O  CM  ”  ^  O  .  O  .  ^  O  O  ”  c 

•  «  •  • 


c 

■6 

< 


CO 

ss 

(S 

h. 

o 

CO 

c 

o 

o 

£ 

9 

o 

o 

c 

o 

ss 

CO 

£ 


111 


APPENDIX  F 


STUDY  MATERIALS; 

Task  Scenarios/Profiles,  Attribute  Definitions,  and  Decision  Questionnaire 
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TASK-  REMOVE/  INSTALL  SAFETY  DEVICES  ON  ENGINE  COMPONENT 

SCENARIO 

Change  which  Precipitated  the  Analysis: 

In  order  to  correct  a  history  of  mishaps  due  to  RX32  engine  component  failures, 
a  new  safety  device  has  been  developed  for  the  future  tactical  fighter  which  has 
proven  to  be  highly  effective.  A  decision  has  subsequently  been  made  to  retrofit  this 
new  safety  device  onto  all  existing  aircraft. 

Personnel: 

The  aircraft  mechanics  responsible  for  performing  this  retrofit  are  all  2nd  term 
enlistees  (i.e.,  they  have  ail  completed  a  three  year  enlistment  and  have  re-enlisted  for 
an  additional  three  more  years.)  Their  general  aptitude  scores  are  in  the  40-80th 
percentile  of  the  ASVAB.  This  is  within  a  standard  deviation  of  the  mean. 

Task: 


installing  the  new  safety  device  is  an  elaborate  procedure  involving  several 
hours  of  time  and  dozens  of  steps.  A  moderate  amount  of  both  mental  and  motor  skills 
are  required.  While  the  number  of  mishaps  resulting  from  a  failure  of  this  component 
is  relatively  low,  the  impact  that  such  a  failure  has  is  relatively  high.  It  is  important  that 
the  safety  device  be  installed  but  the  total  number  of  accidents  which  could  be  avoided 
is  rather  small.  While  the  device  itself  is  rather  simple,  with  only  7  components,  there 
is  also  a  large  amount  of  information  to  consider  to  ensure  that  it  operates  properly. 
The  largest  demand  on  the  aircraft  mechanic  is  integrating  the  device  into  the  existing 
system.  Until  the  entire  inventory  has  been  rotated  through  the  maintenance  facility 
the  mechanics  will  be  scheduled  to  install  a  new  safety  device  approximately  once 
every  week. 

Environment: 

The  installation  of  the  safety  device  is  performed  in  an  aircraft  maintenance 

depot. 

Currently  Available  Training  Resources  and  Aiding  Technologies: 

The  safety  devices  are  installed  exclusively  by  aircraft  maintenance  mechanics 
at  the  maintenance  depot.  The  current  scheduling  load  appears  to  preclude  extensive 
formal  training  to  perform  this  installation  and  the  novelty  of  the  task  has  resulted  in  an 
inherent  lack  of  experts  to  assist  in  OJT.  The  design  contractor  has  supplied  a  series 
of  installation  technical  manuals  for  support. 
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Task  Profile-  Remove/Install  Safety  Devices  on  Engine  Component 


Personnel  Aptitude: 

ASVAB  Gen  Score 

Personnel  Experience: 

Years 

Task  Steps:  Number 

Mental  Skills  Required: 

Motor  Skills  Required; 

Task  Frequency: 

Task  Performance  Rate: 

Task  Criticality: 

Task  Hazards; 

Information  Amount: 

Equip.  Components:  Number 

Training  Resources: 

Aiding  Technology: 


low 


40 


low 


mid 


high  ~1 
80  100 


3+ 


1-3  \  4-9  1  Id- 14 


not  applicable!  low 


not  applicable!  lo^ 


high  I 


high  1 


daily 


not  applicabld 


monthly  I  yearly  I 


mid  I  high  ~1 


not  applicable! 


none  I  low 


high  I 


high  I 


unknown  I  low  |  mid 
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TASK-  TEST  GEAR-BOX  CARBON  SEALS 
SCENARIO 


Change  which  Precipitated  the  Analysis: 

Trend  analysis  of  Air  Force-wide  maintenance  reports  indicate  an  unexpectedly 
high  failure  rate  of  a  new  type  of  gear-box  carbon  seals.  Since  this  seal  design  is 
used  on  several  types  of  operational  aircraft,  a  decision  has  been  made  to  employ  a 
new,  more  stringent  test  of  all  gear-box  seals  on  the  affected  aircraft. 

Personnel: 

The  aircraft  mechanics  typically  expected  to  test  these  seals  have  nine  months 
of  technical  school  and  three  months  of  on-the-job  experience.  Their  general  aptitude 
scores  are  in  the  80-90th  percentile  of  the  ASVAB. 

Task: 


Testing  gear-box  carbon  seals  is  a  laborious  procedure  involving  only  a  handful 
of  steps.  The  test  itself  results  in  a  simple  satisfactory/  unsatisfactory  conclusion  and 
requires  very  little  information  for  the  mechanic  to  consider.  The  main  difficulty  in 
testing  is  physically  accessing  the  seals.  Mechanics  use  a  simple  vacuum-testing 
device  with  only  five  components.  It  is  important  that  the  gear-box  seals  perform 
nominally  since  the  gear-box  is  on  the  critical  path  for  aircraft  operation.The  testing 
procedure  will  be  performed  at  all  air  bases  on  a  rotating  three  month  basis. 

Environment: 

Carbon  seal  testing  is  performed  in  the  hangar. 

Currently  Available  Training  Resources  and  Aiding  Technologies: 

Flightline  maintenance  NCO's  have  been  briefed  on  the  procedures  associated 
with  the  new  test  and  are  responsible  for  training  the  more  junior  personnel  to  perform 
the  test  within  the  current  OJT  curriculum. 
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Task  Profile-  Test  Gear-Box  Carbon  Seals 


TASK  -  ADJUST  DAILY  MAINTENANCE  PLANS 
TO  MEET  OPERATIONAL  COMMITMENTS 

SCENARIO 


Change  which  Precipitated  the  Analysis: 

The  increased  rate  of  operational  sorties  flown  in  the  recent  Gulf  Crisis  has 
placed  great  demands  on  maintenance  activity  schedulers  to  handle  large  numbers  of 
unexpected,  high  priority  aircraft  repairs  while  constrained  by  both  time  and  a  limited 
number  of  experience  maintenance  technicians.  In  effect,  sortie  rate  generation  is 
directly  dependent  upon  the  schedulers'  abilities  to  resolve  these  problems. 

Personnel: 

These  maintenance  schedulers  are  Air  Force  Reservists  whose  experience  with 
daily  maintenance  planning  has  been  limited  to  monthly  weekend  and  annual  week- 
long  service  commitments.  Their  general  aptitude  scores  are  in  the  75-95th  percentile 
of  the  ASVAB.  This  is  a  full  standard  deviation  above  the  mean. 

Task: 


Adjusting  daily  maintenance  plans  to  meet  operational  commitments  involves 
prioritizing  and  scheduling  maintenance,  and  predicting  what  resources  will  be 
needed.  It  is  critical  that  the  most  important  maintenance  tasks  are  accomplished  by 
the  most  experienced  personnel.  Unfortunately,  while  such  a  task  is  typically 
dependent  on  thorough  analyses  of  available  information,  the  reactive  nature  of  the 
combat  environment  avails  itself  to  very  little  predictive  data. 

Environment: 

Adjusting  daily  maintenance  plans  is  performed  in  an  office  environment. 

Currently  Available  Training  Resources  and  Aiding  Technologies: 

A  substantial  portion  of  the  schedulers'  active  duty  assignments  are  targeted  at 
developing  more  proficient  scheduling  skills.  Current  scheduling  procedures  are 
performed  primarily  by  hand. 
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Task  Profile- 

Adjust  Daily  Maintenance  Plans  to  Meet  Operational  Conunittments 


Peiwnnel  Aptitude: 

ASVAB  Gen  Score 

Pervjnnel  Experience: 

Years 

Task  Steps:  Number 

Mental  Skills  Required: 

Motor  Skills  Required: 

Task  Frequency: 

Task  Performance  Rate: 

Task  Oiticality: 

Task  Hazards: 

Information  Amount: 

Equip.  Components:  Number 

Training  Resources: 

Aiding  Technology: 
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mid 


0 
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J 


low  I  mid 
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Tow  I  mid  I  tiigh  I 

mid  I  high  I 

1-3  I  T9  I  mi  nq 

1  none  1  low  THM  Tiiili  1 


i  none' 


mid  I  hiah  1 
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TASK-  ISOLATE  MALFUNCTION  WITH  ENGINE  SYSTEM 

SCENARIO 

Change  which  Precipitated  the  Analysis: 

A  new  electronic  testing  device  has  been  developed  to  assist  aircraft  mechanics 
in  isolating  malfunctions  with  the  internal  fuel  flow  control  system  of  the  F-100  jet 
engine. 

Personnel: 

The  aircraft  mechanics  currently  performing  this  task  are  second  year  enlistees 
(i.e.,  they  have  completed  their  first  three  year  enlistment  and  are  currently  serving  a 
second  three  year  enlistment).  Their  general  aptitude  scores  are  in  the  70-90th 
percentile  of  the  ASVAB.  which  is  approximately  one  standard  deviation  above  the 
mean. 

Task: 


Isolating  fuel  flow  controller  malfunctions  has  five  subtasks  -  formulate  possible 
causes  of  the  malfunction,  remove  suspect  component,  inspect/test  suspect 
component,  replace  component,  and  functionally  test  the  system.  The  most  difficult 
component  of  this  task  is  formulating  possible  causes  of  the  malfunction  which 
correspond  to  the  manifest  symptoms.  Once  the  mechanic  has  postulated  a  cause,  the 
motors  skills  necessary  for  removing/replacing  equipment  components  are  at  a  much 
simpler  level.  The  mechanic  typically  refers  to  technical  specifications  and  repair 
manuals  during  task  performance.  These  documents  are  voluminous.  The  new 
testing  device  provides  more  information  to  the  mechanic  than  before,  but  it  is  fairly 
complicated  to  operate  with  dozens  of  components.  Mechanics  perform  this  task  every 
few  days  and  it  may  require  several  hours  to  isolate  a  particular  malfunction.  The  task 
of  isolating  malfunctions  is  critical  to  the  performance  of  the  aircraft  and  safety  of  the 
crew. 

Environment: 

The  controlled  environment  of  the  aircraft  hangar  (i.e.,  systems  shut-down,  fuel 
drained,  controlled  climate,  etc.)  presents  few,  if  any,  hazards  to  the  mechanic. 

Currently  Available  Training  Resources  and  Aiding  Technologies: 

Currently,  there  are  two  major  training  resources,  several  months  at  a  technical 
school  and  on-the-job  experience.  The  repair  manuals  serve  as  the  only  available 
information  aids. 
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Task  Profile-  Isolate  Malfunction  w/  Engine  System 


Persoonel  Aptitude: 
ASVAB  Gen  Score 


Personnel  Experience: 

Yean 


Task  Steps:  Number 


Mental  Skills  Required: 


Motor  Skills  Required: 


Task  Frequency: 


Task  Performance  Rate: 


Task  Criticality: 


Task  Hazards: 


Information  Amount: 


Equip.  Components:  Number 


Training  Resources: 


Aiding  Technology: 
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TASK  -  EMPLOY  LIFE-SAVING  TECHNIQUES 
SCENARIO 


The  change  which  precipitated  the  analysis: 

The  recent  Gulf  Crisis  has  required  security  police  with  less  experience  to 
handle  tasks  which  normally  were  reserved  for  more  experienced  personnel. 

Personnel: 

The  security  police  have  three  months  of  police  training  and  a  year  of  on-the-job 
experience.  Their  general  aptitude  scores  are  in  the  75-95th  percentile  of  the  ASVAB 
(Armed  Services  Vocational  Aptitude  Battery).  This  is  a  full  standard  deviation  above 
the  mean. 

Task: 


Employing  life-saving  techniques  requires  a  fair  amount  of  mental  skill.  The 
decisions  to  be  made  are  not  especially  difficult  but  the  stress  of  the  situation  requires 
an  amount  of  mental  discipline.  There  are  only  a  handful  of  steps  necessary  for 
rescue  breathing,  CPR,  severe  bleeding,  poisoning,  etc.  The  motor  skills  necessary  to 
save  a  life  are  not  demanding  and  the  opportunity  to  use  the  skills  are  infrequent, 
perhaps  only  once  in  a  year.  The  rate  of  performance  is  high  of  course,  and  the 
criticality  of  the  task  is  very  high.  The  hazards  in  performing  the  task  are  low.  The 
amount  of  information  to  process  is  moderate. 

Environment: 

Employing  life-saving  techniques  is  primarily  performed  at  the  scene  of 
incidents. 

Current  methods  of  support: 

The  following  resources  are  available  for  supporting  this  task.  Security  police 
are  extensively  trained  in  advanced  first  aid  and  life-saving  procedures.  There  are 
good  life-saving  manuals  as  well. 
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Profile  of  81 IXX  Task-  Employ  Life-Saving  Techniques 


Personnel  Aptitude: 

ASVAB  Gen  Score 

Personnel  Experience: 

Years 

Task  Steps:  Number 

Mental  Skills  Required: 

Motor  Skills  Required: 

Task  Frequency: 

Task  Performance  Rate: 

Task  Criticality: 

Task  Hazards: 

Information  Amount: 

Equip.  Components:  Number 

Training  Resources: 

Aiding  Technology: 


I  low  I  rriI3~ 

0  40 


80  100 


0  1 
I - TT 


I  not  applicable!  low 

(not  applicable  I 


3  3+ 

TO-R-  I  ~15q 


high  I 


yc.irly 


mid  I  high  | 

I  daily  |  weekly  |  monthly 

I  not  applicabid  low  |  mid 

I  not  applicablel  low  I  mid  ~ 

mid  I  high  j 

unknown  I  low  RigE  | 


I  none 


I  none  I  low  |  mid 
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TASK  -  CONDUCT  ANTI-ROBBERY  TRAINING 


SCENARIO 


The  change  which  precipitated  the  analysis: 

A  recent  surge  in  robberies  in  base  housing  has  caused  the  base  commander 
to  provide  anti-robber  training  for  all  interested  personnel.  The  base  security  police 
have  been  assigned  the  task  of  conducting  the  training. 

Personnel: 

Senior  security  personnel  have  been  selected  to  provide  the  training.  Most  of 
them  have  ten  years  or  more  experience.  Their  general  aptitude  scores  are  in  the  40- 
80th  percentile  of  the  ASVAB  (Armed  Services  Vocational  Aptitude  Battery).  This  is 
within  a  standard  deviation  of  the  mean. 

Task: 


Conducting  anti-robbery  training  requires  a  moderate  amount  of  mental  skill  but 
no  specific  motor  skills.  The  training  courses  are  to  be  offered  once  weekly  for  a 
month.  The  actual  task  of  training  requires  responsiveness  to  questions  and  the 
ability  to  process  a  moderate  amount  of  information. 

Environment: 

Anti-robbery  training  is  conducted  in  a  classroom  environment. 

Current  methods  of  support: 

The  following  resources  are  available  for  supporting  this  task.  There  is  little 
formal  training  for  personnel  conducting  anti-robber  training  but  there  are  numerous 
visual  aids,  including  a  training  film  about  how  to  protect  your  house  from  robbers. 
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Profile  of  81 IXX  Task-  Conduct  Anti-Robbery  Training 


Personnel  Aptinidc: 

ASVAB  Gen  Score 

Personnel  Experience: 

Yean 

Task  Steps:  Number 

Mental  Skills  Required: 

Motor  Skills  Required: 

Task  Frequency: 

Task  Performance  Rate: 

Task  Criticality: 

Task  Hazards: 

Information  Amount: 

Equip.  Components:  Number 

Training  Resources: 

Aiding  Technology: 
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TASK-  WRITE  SECURITY  POLICE  COMPUTER  PROGRAMS 

SCENARIO 

The  change  which  precipitated  the  analysis: 

Technological  changes  have  greatly  influenced  police  work.  The  most 
pervasive  has  been  the  introduction  of  computers.  With  computers  come  the  necessity 
for  computer  programmers  to  write  the  programs. 

Personnel: 

The  policemen  we  are  considering  have  completed  a  three  year  enlistment  and 
have  re-enlisted  for  three  more  years.  Their  general  aptitude  scores  are  in  the  40-80th 
percentile  of  the  ASVAB  (Armed  Services  Vocational  Aptitude  Battery).  This  is  within  a 
standard  deviation  of  the  mean. 

Task: 


Programming  computer  programs  is  a  complex  task  involving  dozens  of 
substeps.  Each  of  the  steps  are  comparatively  easy  but  the  integration  of  program 
functions  is  moderately  difficult.  Computer  programs  are  written  and  need  modification 
on  a  weekly  basis.  Most  of  the  programs  written  are  for  accounting  and  payroll 
activities  which  are  not  life  or  mission  critical.  There  is  a  large  amount  of  information 
which  must  bo  taken  into  account  to  write  a  program,  information  about  syntax, 
integration  and  the  information  which  is  to  be  processed.  The  programs  are  written  on 
a  standard  computer  keyboard  which  is  similar  to  a  typewriter  keyboard. 

Environment: 

Writing  computer  programs  is  performed  in  an  office  environment. 

Current  methods  of  support: 

The  following  resources  are  available  for  supporting  this  task.  Security  police 
receive  little  training  and  little  aiding  in  this  task.  Most  of  the  personnel  which  perform 
this  task  have  received  training  in  high  school  or  college. 
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Profile  of  811 XX  PSTask-  Write  Security  Police  Computer  Programs 


Personnel  Aptitude: 


low 


mill 


high  I 


ASVAB  Gen  Score 

Personnel  Experience: 

Years 

Task  Steps:  Number 

Mental  Skills  Required: 

Motor  Skills  Required: 

Task  Frequency; 

Task  Perfwmance  Rate: 

Task  Criticality: 

Task  Hazards: 

Information  Amount: 

Equip.  Components:  Number 

Training  Resources: 

Aiding  Technology: 


40 


80  100 


I  not  applicable!  low 
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I  none 
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TASK-  EVALUATE  SITUATIONS  AT  INCIDENT  SCENES 

SCENARIO 

The  change  which  precipitated  the  analysis: 

There  have  been  a  number  of  dangerous  situations  which  have  arisen  because 
inexperienced  security  police  have  evaluated  situations  at  incident  scenes  incorrectly. 
A  new  directive  has  been  issued  that  require  a  senior  policeman  to  be  at  the  scene  of 
all  violent  incidents. 

Personnel: 

The  policemen  we  are  considering  have  completed  a  three  year  enlistment  and 
have  re-enlisted  for  three  more  years.  Their  general  aptitude  scores  are  in  the  40-80th 
percentile  of  the  ASVAB  (Armed  Services  Vocational  Aptitude  Battery).  This  is  within  a 
standard  deviation  of  the  mean. 

Task: 


Evaluating  situations  at  incident  scenes  requires  a  dozen  of  steps  to  be  quickly 
followed.  There  is  a  moderate  amount  of  both  mental  and  motor  skills  required. 
Violent  incidents  occur  on  a  weekly  basis.  It  is  important  that  the  safety  of  the  police  is 
guarded  by  experienced  personnel  who  know  what  to  look  for.  There  is  a  moderate 
amount  of  information  available  tor  assimilation  at  any  incident.  The  task  is  highly 
critical  as  the  safety  of  security  personnel  is  at  stake.  There  are  many  potential 
physical  hazards  in  a  violent  incident  scene,  some  life-threatening.  Security  police 
have  megaphones,  radios,  weapons,  etc.  to  handle  serious  incidents. 

Environment: 

Situation  evaluation  is  conducted  at  the  incident  scene. 

Current  methods  of  support: 

The  following  resources  are  available  for  supporting  this  task.  There  is  an 
extensive  training  school  for  evaluating  incidents.  Many  incidents  are  enacted  at  the 
schools  in  a  very  realistic  manner.  Security  police  can  communicate  via  radio  to  more 
experienced  personnel  until  they  arrive. 
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Profile  of  81 1 XX  Task-  Evaluate  Situations  at  Incident  Scenes 


Personnel  Aptitude:  1 

ASVAB  Gen  Score  0 


80  100 


Personnel  Experience:  L 

Years  0 


Task  Steps;  Number 


Mental  Skills  Required: 


Motor  Skills  Required: 


Task  Frequency: 


Task  Performance  Rate: 


Task  Qidcality: 


Task  Hazards: 


Information  Amounu 


Equip.  Components:  Number  I  not  a 


Training  Resources: 


Aiding  Technology; 
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ATTRIBUTE  DEFINITION  SHEET 


Personnel  Aptitude:  Aptitude  is  defined  as  the  natural  capacity,  ability,  or  suitability 
of  an  individual  to  learn  a  specified  task.  In  each  of  the  following  scenarios,  the 
Armed  Services  Vocational  Aptitude  Battery  (ASVAB)  score  represents  a 
composite  measure  of  aptitude  types  (i.e.,  mechanical,  electrical,  etc.).  The  mean 
score  is  59  with  a  standard  deviation  of  1 7. 

Personnel  Experience:  Experience  represents  the  knowledge  and  skills  derived 
from  previous  performance  the  task.  It  can  be  acquired  through  both  practice 
and/or  operational  performance. 

Task  Steps:  The  number  of  individual  steps  required  to  perform  the  task. 

Mental  Skills  Required:  The  mental  skills  required  is  a  measure  of  the  complexity 
of  the  decisions  involved  with  the  performance  of  this  task.  For  example,  if  there 
are  no  decisions  to  be  made  (i.e.,  a  fixed  sequence  of  steps  is  required)  mental 
skills  required  is  not  applicable.  For  simple  decisions  (i.e.,  yes/no),  the  mental 
skills  required  is  low.  And,  for  decisions  which  are  based  on  evaluating 
numerous,  inter-related  conditions,  the  mental  skills  required  is  high. 

Motor  Skills  Required:  The  motor  skills  required  refers  to  the  difficulty  of  the 
physical  manipulations  necessary  to  install,  remove,  adjust,  align,  and  measure 
the  components  of  the  task  equipment. 

Task  Frequency:  Task  frequency  refers  to  how  often  or  frequently  this  task  is 
performed  on  the  job. 

Task  Performance  Rate:  Performance  rate  is  the  speed  at  which  the  task  must  be 
accomplished.  A  major  implication  of  performance  rate  is  whether  the  person  has 
time  to  refer  to  an  aid  while  he/she  is  performing  the  task. 

Task  Criticality:  Task  criticality  is  a  measure  of  the  importance  of  performing  a  given 
task  in  meeting  some  goal.  Criticality  is  gauged  by  the  consequences  of 
inadequate  performance  or  nonperformance  (i.e.,  repair  expense,  loss  of  time, 
etc.). 

Task  Hazards:  Task  hazards  are  categorized  into  two  types,  personnel  and 
equipment  hazards.  Personnel  hazards  are  characteristics  of  the  tasks  which,  if 
improperly  performed,  can  cause  injury  or  death  to  the  performer  or  others. 
Equipment  hazards  are  characteristics  of  the  tasks  which,  if  improperly 
performed,  can  cause  various  levels  of  damage  which  potentially  threaten  task 
accomplishment. 
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Information  Amount:  The  quantity  of  information  the  person  must  process  in  order 
to  perform  the  task. 

Equipment  Components:  The  number  of  components  in  the  equipment  the 
operator/maintainer  employs  to  perform  the  task. 

Training  Resources:  Training  resources  include  the  books,  instructors,  classrooms, 
educational  software,  simulators,  etc.  which  prepare  the  person  for  task 
performance  before  they  do  the  task. 

Aiding  Technology:  Aiding  technology  refers  to  the  illustrations,  tables,  flowcharts, 
checklists,  schematics  etc.  which  support  the  person's  performance  as  they  do 
the  task. 
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Task:  Adjust  Daily  Maintenance  Plans  to  Meet  Operational  Commitments  (B28-1) 
Personnel:  Jet  Engine  Maintenance  Technicians  (454X0) 


PART  I.  AIDING  SOLUTION  SPECIFICATION 


1.  Can  aiding  techniques  be  employed  to  help  overcome  this  problem?  (Check  one) 

_ Yes  (Skip  Question  2,  go  to  Question  3) 

_ No  (Go  to  Question  2) 

2.  If  aiding  techniques  cannot  be  employed  to  help  resolve  this  problem,  what  is  it  that 
prevents  their  use?  (Check  one  and  proceed  directly  to  Question  6) 

_ Principle 

_ Practice 

_ Policy 


Questions  3  and  4.  Given  a  finite  level  of  aiding  resources,  how  would  you  allocate 
them  among  the  following  aiding  approaches  and  aiding  techniques  to  most 
satisfactorily  resolve  this  aiding  problem?  (Specify  relative  percentages  of  resources 
for  each  applicable  approach/technique:  TQTAL  »  1 0Q  %.) 


3.  If  aiding  can  be  employed  to  help  resolve  this  problem,  what  approach(es)  to  aiding 
would  you  use? 

_ %.  Display  Information 

_ %.  Monitor  Performance 

_ %.  Assist  Execution 


4.  How  would  you  allocate  resources  among  the  following  aiding  techniques  to 
support  the  approach(es)  selected  above? 

_ 2k  Information  2k  Heuristic 

_ 2k  Procedural  2k  Assistive 

_ 2k  Algorithmic 


5.  Would  this  aiding  solution  alone  be  sufficient  to  resolve  the  problem,  independent 
of  any  training  solution? 

Yes 


No 


PART  II.  TRAINING  SOLUTION  SPECIFICATION 

6.  Can  training  techniques  be  employed  to  help  overcome  this  problem? 

_ Yes  (Skip  Ques.  7.  go  to  Ques.  8) 

_ No  (Go  to  Ques.  7) 

7.  If  training  techniques  cannot  be  employed  to  resolve  this  problem,  what  is  it  that 
prevents  their  use?  (Check  one  and  proceed  directly  to  the  next  Task  Scenario. 
OO  NOT  answer  any  of  the  remaining  questions.) 

_ Principle 

_ Practice 

_ Policy 

Questions  8  and  9.  Given  a  finite  level  of  training  resources,  how  would  you  allocate 
them  among  the  following  training  approaches  and  training  techniques  to  most 
satisfactorily  resolve  this  problem?  (Specify  relative  percentages  of  resources  for 
each  applicable  approach/technique:  TOTAL  «  IQQ  %.) 

8.  If  training  can  be  employed  to  resolve  this  problem,  what  approach(es)  to  training 
would  you  use? 

_ %,  Presentation 

_ Modeling 

_ %,  Coaching 

_ %,  Exploration: 

9.  How  would  you  allocate  resources  among  the  following  training  techniques  to 
support  the  approach(es)  selected  above? 

_ %,  Classroom 

_ Independent  Study 

_ Simulators 

_ 2k  Job  Experience 
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10.  Would  this  training  solution  alone  be  sufficient  to  resolve  the  problem, 
independent  of  any  aiding  solution? 

Yes 


No 


PART  III.  COMBINED  SOLUTION  SPECIFICATION 


1 1 .  Given  a  finite  level  of  aiding  and  training  resources,  how  would  you  allocate  them 
among  the  following  aiding/training  approaches  to  most  satisfactorily  resolve  this 
problem? 


,2L.Aiding  Approaches 
^Training  Approaches 
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